Mechanism for Dynamically Allocating Physical Storage Device Resources in a Virtualized Environment

Abstract

A storage device is disclosed. The storage device may include a storage means for data and at least one input / output (I / O) queue for requests from at least one virtual machine (VM) on a host device. The storage device may support an I / O queue creation command to request allocation of an I / O queue for a VM. The I / O queue creation command may include an LBA range attribute of a range of logical block addresses (LBAs) to be associated with the I / O queue. The storage device may map the range of LBAs to a range of physical block addresses (PBAs) in the storage means.

Description

[0001] Related Application Data

[0002] This application claims the benefit of U.S. Provisional Patent Application Ser. No. 62 / 642,596, filed Mar. 13, 2018, which is incorporated herein by reference in its entirety for all purposes.

[0003] This application is related to U.S. patent application Ser. No. 15 / 959,108, filed Apr. 20, 2018, now pending, which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0004] The present inventive concept generally relates to storage devices, and more particularly, to supporting access to storage devices by virtual machines that may be isolated from each other. Background Art

[0005] Single Root Input / Output Virtualization (SR-IOV) is a specification-supported interface mechanism that allows a single physical Peripheral Component Interconnect Express (PCIe) device to appear as multiple separate physical PCIe devices. For performance and manageability reasons, SR-IOV helps share and isolate PCIe resources while promoting interoperability.

[0006] SR-IOV has been around for a decade for network adapters. More recently, SR-IOV has started to include storage. Central Processing Unit (CPU) processing has provided resource isolation, which has helped the rapid virtualization adoption with a hypervisor as the master and Virtual Machines (VMs) as the slaves. Using SR-IOV, network and storage devices expose Physical Function (PF) and Virtual Function (VF) devices. Together, these provide device isolation sufficient to transform a physical server into multiple virtual servers such that all applications can run in their own isolated spaces.

[0007] Although compute processing, networking, and storage devices form the three virtualization pillars, storage devices and storage device vendors still lag in compliance with SR-IOV. This fact may be because, unlike networking, storage devices define a data address space referenced by a range of logical block addresses (LBAs). This LBA range can only be subdivided into a limited number of units. Additionally, storage devices require physical hardware gates to support additional VFs because VFs are hardware functions directly exposed to the virtual machine's (VM) Peripheral Component Interconnect (PCI) space. Adding SR-IOV to a storage / network device increases its gate count and chip size and consumes more power.

[0008] SR-IOV solves the hardware isolation problem while providing bare-metal performance because, unlike para-virtualized devices, I / O does not have to go through the hypervisor. Non-Volatile Memory Express (NVMe) storage devices are the latest to adopt SR-IOV. However, for storage devices, there may be other mechanisms to provide isolated access for multiple VMs.

[0009] There is still a need for a method to provide functionality similar to that provided by SR-IOV but without the hardware requirements and limitations imposed by SR-IOV. SUMMARY

[0010] One aspect of the present disclosure provides a storage device for dynamically allocating physical storage device resources in a virtualized environment, including: a storage device for data; and at least one input / output (I / O) queue for requests from at least one virtual machine (VM) on a host device, wherein the storage device supports an I / O queue creation command to request an I / O queue for the at least one VM of the at least one VM, the I / O queue creation command including an LBA range attribute of a range of logical block addresses (LBAs) to be associated with the I / O queue, and, wherein the storage device maps the range of LBAs to a range of physical block addresses (PBAs) in the storage device for data.

[0011] Another aspect of the present disclosure provides a field-programmable gate array (FPGA) for dynamically allocating physical storage device resources in a virtualized environment, including: at least one virtual input / output (I / O) queue for requests from at least one virtual machine (VM) on a host device; and mapping logic for mapping the virtual I / O queues of the at least one virtual I / O queue to I / O queues on a storage device such that I / O requests received from the VM in the virtual I / O queue are passed to the storage device via the I / O queue, and results received from the storage device in the I / O queue are passed to the VM via the virtual I / O queue, wherein the FPGA supports a virtual I / O queue creation command to request allocation of the virtual I / O queues of the at least one virtual I / O queue for the VM of the at least one VM, the virtual I / O queue creation command includes an LBA range attribute of a range of logical block addresses (LBAs) to be associated with the virtual I / O queue, and, wherein a storage device separate from but connected to the FPGA maps the range of the LBAs to a range of physical block addresses (PBAs) in the storage device.

[0012] Another aspect of the present disclosure provides an article including a non-transitory storage medium for dynamically allocating physical storage device resources in a virtualized environment, the non-transitory storage medium storing instructions that, when executed by a machine, cause: receiving a first request from a virtual machine (VM) on a host device, the first request going to a storage device; capturing the first request to prevent it from reaching the storage device; sending a second request to a field-programmable gate array (FPGA), the second request simulating the first request; receiving a result of the second request from the FPGA; and sending the result of the second request to the VM. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Devices supporting isolated virtual machine (VM) access to a storage device according to embodiments of the inventive concept are shown.

[0014] Figure 2 Shows Figure 1 additional details of the device.

[0015] Figure 3 Shows Figure 1 the communication path between the VM of Figure 1 and the storage device of Figure 1 wherein the storage device of

[0016] Figure 4 Shows Figure 1 the communication path between the VM of Figure 1 and the storage device of Figure 1 The storage device exposes multiple physical functions.

[0017] Figure 5 shows Figure 1 details of the storage device.

[0018] Figure 6 shows the Figure 1 extended I / O queue creation command for the storage device.

[0019] Figure 7 shows the physical storage device of the Figure 1 storage device divided into multiple namespaces.

[0020] Figure 8 shows Figure 1 memory mapping of the doorbell in the storage device to support VM isolation.

[0021] Fig. 9 shows the Figure 3 extended virtual I / O queue creation command for the Field Programmable Gate Array (FPGA).

[0022] Fig.10 shows the virtual I / O queue that supports mapping to the Figure 1 I / O queue in the storage device of the Figure 3 FPGA.

[0023] Fig.11 shows an example process flowchart for Figure 1 the storage device to allocate I / O queues for VMs according to an embodiment of the inventive concept.

[0024] Fig.12 shows an example process flowchart for Figure 3 the FPGA to allocate virtual I / O queues for VMs according to an embodiment of the inventive concept.

[0025] Fig.13 shows an example process flowchart for Figure 3 the hypervisor to process control requests from Figure 3 the virtual machines according to an embodiment of the inventive concept.

[0026] Fig.14 shows an example process flowchart for Figure 1 the storage device or Figure 3 the FPGA to map the memory address of the doorbell to different operating system pages to support VM isolation according to an embodiment of the inventive concept. DETAILED DESCRIPTION

[0027] Reference will now be made in detail to embodiments of the inventive concept, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the inventive concept. However, it will be understood that the inventive concept may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0028] It should be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the inventive concept, a first module may be referred to as a second module, and similarly, a second module may be referred to as a first module.

[0029] The terms used in the description of the inventive concept herein are for the purpose of describing particular embodiments only and are not intended to limit the inventive concept. As used in the description of the inventive concept and the appended claims, the singular forms "a," "an," and "the" are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should also be understood that when used in this specification, the terms "comprise" and / or "comprises" specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The components and features of the drawings are not necessarily drawn to scale.

[0030] Currently, early adopters of single root input / output virtualization (SR-IOV) for non-volatile memory express (NVMe) storage devices have taken the route of providing a limited number of virtual functions (VFs) and associated namespaces per physical function (PF). The allocation of the logical address space range depends on the namespace allocation, and the direct bare-metal access to a VM depends on the number of supported VFs. This implementation creates problems for supporting large-capacity storage devices because it must support additional hardware at the cost of power and die size to meet the performance requirements for directly supporting multiple VMs. At a minimum, supporting the SR-IOV function requires the device to provide a separate peripheral component interconnect (PCI) configuration space, I / O bar, I / O queues for submission and completion queues, message signaled interrupt (MSI-X), and the doorbell address for the supported queues for each VF.

[0031] The deficiencies of the fixed allocation method available in the first SR-IOV storage device have prompted the NVMe committee to develop another specification that eliminates implementation restrictions, such as changing fixed resource allocation to dynamic resource allocation and management. Many currently open and actively proposed changes are attempting to address the resource allocation problem. However, they are constrained by physical device limitations, such as the supported VFs, which increases the inevitable specification complexity.

[0032] Although there are a few leading hypervisors in the market, such as VMWare, Microsoft, Citrix XenServer, KVM, Qemu, and Oracle VM, currently only a few hypervisors from this set are actively adapting to market changes. By doing so, they have captured the majority of the market share. Due to the nature of their programming environments, each hypervisor environment can be considered a custom implementation. Therefore, supporting changes to this implementation may be important.

[0033] Embodiments of the inventive concept define a simpler mechanism to program SR-IOV storage devices to provide VM isolation and performance. Embodiments of the inventive concept follow a simpler approach by relying on the hypervisor to map resources. Embodiments of the inventive concept may include policies for:

[0034] 1) Extend performance to the VM by providing direct application access to the I / O queues (submit and complete queue pairs) of the VM.

[0035] 2) Directly map the capabilities of the I / O queues (submit and complete queues) to the VM.

[0036] 3) Encapsulate the controller details in the hypervisor for the management path.

[0037] 4) Dynamically isolate the physical resources required for each separate VM.

[0038] Embodiments of the inventive concept can effectively provide SR-IOV type functionality by adding new NVMe storage device support as a feature.

[0039] Embodiments of the inventive concept define the following new features:

[0040] 1) An advanced management mechanism for directly remapping the NVMe I / O submit queue and the I / O completion queue (collectively referred to as the I / O queue pair) to the VM for performance purposes.

[0041] 2) A mechanism in the hypervisor for implementing virtualized controllers that dynamically maps hardware I / O queues to the VM as a set of virtualized controllers.

[0042] 3) A mechanism for mapping a logical address space to a logical unit or a namespace.

[0043] 4) A mechanism for mapping additional I / O queues that are not available in a storage device using an additional method.

[0044] 5) A method for providing VM-specific isolation for shared resources.

[0045] Embodiments of the inventive concept provide the following advantages over the prior art:

[0046] 1) Provide SR-IOV-like functionality without expensive SR-IOV hardware requirements.

[0047] 2) Provide additional I / O resource isolation and performance advantages for VMs beyond device specifications.

[0048] 3) Provide hardware capabilities that can be fully virtualized to avoid changes to the built-in drivers of the operating system (O / S).

[0049] 4) Create a Quality of Service (QoS) channel between the storage device and the hypervisor.

[0050] 5) Simplify the hardware requirements for storage virtualization.

[0051] Embodiments of the inventive concept provide a mechanism for directly remapping storage device I / O resources to a VM. Embodiments of the inventive concept use existing remapping resources for memory and interrupts, including an Input-Output Memory Management Unit (IOMMU) for the x86 architecture and additional hardware resources, to map to a large number of VMs that are not typically supported by a storage device.

[0052] To achieve these advantages, a storage device (which may include a Solid State Drive (SSD)) should:

[0053] 1) Support extended I / O queue creation attributes.

[0054] 2) Support simple logical address remapping at the I / O queue level.

[0055] 3) Support doorbells at the O / S page boundary for VM security.

[0056] 4) Publish these extended attributes through one or more fields in accordance with the NVMe specification standard.

[0057] 5) Optionally provide an additional queue priority level arbitration mechanism instead of applying QoS by default.

[0058] Map the NVMe I / O queue pair directly to the VM

[0059] The I / O submission queue and the I / O completion queue can be collectively referred to as an I / O queue pair because they work together and, when resources are limited, a 1:1 mapping is automatically applied. Most, if not all, built-in NVMe drivers create a 1:1 mapping of these queues. Embodiments of the inventive concept target this usage because these are device driver implementations running in the VM.

[0060] A storage device supporting embodiments of the inventive concept can provide extended I / O create commands for the submission and completion queues. The commands can be applied through separate opcodes, which can be marked as optional or can be defined as vendor-specific commands. The extended commands can support the basic NVMe queue create command details defined by the NVMe specification, such as queue size, queue identifier, queue priority level, whether the queue buffer is physically contiguous, interrupt vector, and interrupt enable fields. But in addition to these, the extended commands can also support logical block addressing offsets within the address space of the device. These mechanisms work as follows:

[0061] 1) When a VM is created, the hypervisor exposes a virtualized NVMe storage device in the PCI space of the VM. The device can initially be fully virtualized so that client exits occur when accessing its PCI configuration and I / O memory mapped regions. For interrupts, the hypervisor can set specific MSI-X interrupts to directly interrupt the client VM as needed, according to the hypervisor's implementation.

[0062] 2) In the I / O memory mapped space, the hypervisor can capture any NVMe configuration space changes and virtualize access requests as needed. The hypervisor can expose doorbells at the O / S page level granularity as part of the I / O memory so that the hypervisor can map each doorbell at the VM level. The storage device can also support this function.

[0063] 3) When the VM creates an I / O submission queue, the hypervisor captures the request and maps it to the physical I / O submission queue of the storage device using the "Extended Create I / O Submission Queue" command. The hypervisor and the storage device can use the values provided by the VM and change / add the following:

[0064] a) Map the queue memory to its (multiple) physical pages so that the storage device can access it directly. This mechanism can be provided by the IOMMU on Intel x86-based architectures.

[0065] b) Add queue priorities (if supported), which prioritize this queue relative to other VMs.

[0066] c) Bind the previously created I / O completion queue ID in the VM's space to this submission queue for the storage device I / O queue.

[0067] d) Add mapping of a portion of the physical address space to the start and end values of the logical block address of the VM.

[0068] e) Apply appropriate QoS requirements for minimum or maximum I / O operations, or minimum or maximum bytes transferred per second granularity.

[0069] f) Additionally, if the VM requires it, the storage device can provide global namespace access rights. These global namespace access rights can be specified in an array listing the namespace IDs.

[0070] g) Additionally, if the storage device provides such support, the storage device can provide rights to the type of namespace access, such as read-only, read-write, exclusive access.

[0071] 4) The hypervisor may also capture I / O completion queue creation requests and instead:

[0072] a) Map the queue memory to its (multiple) physical pages so that the storage device can access it directly. On Intel x86-based architectures, the IOMMU can provide this mechanism.

[0073] b) Map the interrupt vectors provided between the actual storage device vectors and the VM through system architecture mechanisms already in place for virtualization (such as the IOMMU).

[0074] 5) Depending on the complexity of the VMs being managed and the hypervisor implementation, embodiments of the inventive concept may map multiple I / O creation requests to a single physical queue. If implemented, the FPGA can handle the I / O completion queue and can interrupt back to the VM guest. This mechanism can address the dynamic queue allocation mechanism.

[0075] 6) In another use of the dynamic queue allocation mechanism, the hypervisor can expose only the required I / O queues based on the VM Service Level Agreement (SLA).

[0076] Once established, the hypervisor-assisted mechanism that can dynamically map hardware I / O queues to VMs can provide the necessary isolation and performance advantages similar to those provided by SR-IOV, but with lower manufacturing, testing, and debugging costs. The hypervisor may also require some changes, but these changes are self-contained and limited to the available hypervisor, thus reducing the overall impact. If a storage device is installed in a system that does not support virtualization, the storage device should operate as a regular NVMe storage device.

[0077] Management address space isolation

[0078] NVMe can share a single, unique logical address space by using namespaces that work like Small Computer Systems Interface (SCSI) Logical Unit Numbers (LUNs). Given a namespace ID, a logical address block can be offset by the address where the namespace starts in the overall logical address mapping.

[0079] In traditional storage devices, the physical address space can be subdivided into logical units by creating namespaces and their coordinates. In traditional storage devices, this subdivision may require additional namespace support to create multiple namespaces. Embodiments of the inventive concept bypass this requirement by attaching the created I / O queues directly to the logical unit space. For example, extended attributes can be defined as part of the logically addressable space of the I / O queue, so the default namespace maps to it. This modification will align each I / O queue so that the VM can directly access the appropriate space. Typically, a VM either requests to access only a private namespace or requests to access a shared namespace. Extended attributes in I / O queue creation can specify the start and end LBAs of the default namespace relative to the global physical address space. This mechanism addresses the hardware namespace management requirement. Any incoming I / O request must pass through the I / O queue. Since the I / O queue already maintains the default namespace mapping offset, it can directly use this programmed offset to translate the LBA address.

[0080] If access to multiple namespaces is required, extended I / O queue creation may have an additional definition to support global namespace access. For example, I / O queue 23: Global namespace accessed: 3.

[0081] Mechanism for mapping additional I / O queues not available in the storage device

[0082] This mechanism involves using additional field-programmable gate array (FPGA) logic that provides I / O queues in a separate space. Using this mechanism, embodiments of the inventive concept can support significantly more I / O queues that can be directly mapped to VMs. The FPGA can support the NVMe specification or a subset of the specification, and arbitrate the mechanism through this specification.

[0083] Full specification support : In this mechanism, the FPGA can fully mimic the NVMe specification and provide full specification-level functionality. The FPGA can provide a memory mapping for associated I / O queue doorbells at the O / S page granularity, provide support for additional I / O queues not available in the device, provide full MSI-X interrupt support for the supported I / O queues, and provide a mapping of the logical address space for each I / O queue. In this mechanism, the storage device does not need to support extended I / O queue functionality at all, which can be fully implemented in additional programmable hardware.

[0084] Some specifications support : If it can be predicted that the number of VMs is not greater than the number of I / O queues supported by the storage device, some or all of the functionality can be implemented within the storage device. The FPGA can then be used as a "pass-through" device for I / O requests from the VMs, or use a one-to-one mapping of virtual I / O queues to storage device I / O queues. The FPGA can still be used to reduce the amount of hardware for implementing functionality in the storage device.

[0085] In either case, the FPGA logic can provide the necessary isolation granularity for each VM. To provide a large number of I / O queues, the FPGA can map many of its exposed I / O queues to a single storage device I / O queue.

[0086] The FPGA can provide isolation of the logical address space mapping structure and the associated class namespace. The FPGA can also provide the necessary MSI-X interrupt mapping capabilities to make the device fully operational.

[0087] Quality of Service

[0088] When multiple VMs perform I / O operations accessing a single storage device, due to the blender effect, they will inhibit performance. When sharing a device at the VM level without isolation, the storage device cannot access a specific VM. For SR-IOV, isolation is provided at the PF mapping level, but its advantages are not published or unknown. Embodiments of the inventive concept can support binding I / O queue resources to a VM, which not only provides natural isolation at the resource level but also to the VMI / O requests in a stream. By configuring the I / O queue identifier, the storage device has complete VM knowledge. If the storage device supports additional priority level arbitration mechanisms (such as those defined in the NVMe specification or as part of vendor-specific commands), the hypervisor can apply these mechanisms to the I / O queue at creation time. The hypervisor can choose to apply these different priority level supports based on the requirements of the VM.

[0089] Based on the provided storage device capabilities, embodiments of the inventive concept can also expose additional fields in the extended I / O queue creation command that provide performance limits or minimum services required for the storage device. The required performance limits or minimum services can be quantified in read / write I / O counts or bytes transferred. Such required performance limits or minimum services can also be settable options based on device support and applied by the hypervisor to the VM.

[0090] Typical usage of embodiments of the inventive concept can be directly applied to storage virtualization in the enterprise segment that makes extensive use of virtual machines (VMs).

[0091] Figure 1 A device for supporting isolated virtual machine (VM) access to a storage device according to an embodiment of the inventive concept is shown. In Figure 1 it, device 105 is shown, which can also be referred to as a host computer or host device. Device 105 can include a processor 110. Processor 110 can be any kind of processor: for example, an Intel Xeon, Celeron, Itanium, or Atom processor, an AMD Opteron processor, an ARM processor, etc. Although Figure 1 a single processor 110 in device 105 is shown, device 105 can include any number of processors, each processor can be a single-core or multi-core processor, and they can be mixed in any desired combination. Processor 110 can run a device driver 115, which can support access to the storage device 120: different device drivers can support access to other components of device 105.

[0092] Device 105 may also include a memory controller 125, which may be used to manage access to the main memory 130. The memory 130 may be any type of memory, such as flash memory, dynamic random access memory (DRAM), static random access memory (SRAM), persistent random access memory, ferroelectric random access memory (FRAM), or non-volatile random access memory (NVRAM) such as magnetoresistive random access memory (MRAM). The memory 130 may also be any desired combination of different memory types.

[0093] Although Figure 1 Device 105 is depicted as a server (which may be a stand-alone or rack-mounted server), embodiments of the inventive concept may include any desired type of device 105 without limitation. For example, device 105 may be replaced with a desktop or laptop computer or any other device that may benefit from embodiments of the inventive concept. Device 105 may also include dedicated portable computing devices, tablet computers, smart phones, and other computing devices.

[0094] Figure 2 is shown Figure 1 additional details of device 105. In Figure 2 Typically, device 105 includes one or more processors 110, which may include a memory controller 125 and a clock 205, which may be used to coordinate the operation of the components of device 105. The processor 110 may also be coupled to the memory 130, which may include, by way of example, random access memory (RAM), read-only memory (ROM), or other state-holding media. The processor 110 may also be coupled to a storage device 120 and a network connector 210, which may be, for example, an Ethernet connector or a wireless connector. The processor 110 may also be connected to a bus 215, to which a user interface 220 and input / output interface ports that may be managed using an input / output engine 225 and other components may be attached.

[0095] Figure 3 is shown Figure 1 of the VM and Figure 1a communication path between storage devices 120, where Figure 1 the storage device 120 only exposes one physical function. In Figure 3 it, three VMs 305-1, 305-2, and 305-3 are shown, which can be Figure 1 instantiated on the host device 105 of Figure 3 Although three VMs 305-1, 305-2, and 305-3 are shown, embodiments of the inventive concept may include a host device 105 that supports any number of VMs. Figure 1

[0096] VMs 305-1, 305-2, and 305-3 can communicate with the hypervisor 310. The hypervisor 310 can create, manage, and run VMs 305-1, 305-2, and 305-3. The hypervisor 310 is typically implemented as software running on the processor 110 of the host device 105 of Figure 1

[0097] To enable interaction with hardware devices, especially hardware that implements single root input / output virtualization (SR-IOV), such hardware devices may expose various physical functions. For example, Figure 3 a field programmable gate array (FPGA) 315 that exposes one physical function (PF) 320 is shown.

[0098] To enable VMs 305-1, 305-2, and 305-3 to interact with hardware devices, various virtual functions (VFs) may also be exposed. VMs 305-1, 305-2, and 305-3 can interact with VFs 325-1, 325-2, and 325-3 instead of having to interact directly with the hardware. VFs 325-1, 325-2, and 325-3 provide a virtualized version of PF 320, enabling VMs using different operating systems (O / S) to effectively interact with the underlying hardware device (e.g., using native O / S drivers). Although Figure 3 three VFs of PF 320 are shown, embodiments of the inventive concept may include any number of VFs per PF; and if the FPGA 315 (or storage device 120) exposes more than one PF, each PF may have a different number of VFs. Each exposed VF may require some hardware support from the underlying device. For example, each VF exposed for the storage device 120 may require additional endtraps and doorbells, which requires additional hardware implementation.

[0099] ​​As described above, PF 320 can be exposed by FPGA 315 instead of by storage device 120, which is the underlying hardware implementing PF 320. FPGA 315 can interrogate storage device 120 to determine which (if any) PFs are exposed by storage device 120, and then FPGA 315 itself exposes comparable PFs that can be directly mapped across FPGA 315 to the corresponding PFs of storage device 120.

[0100] Alternatively, PF 320 can be directly exposed by storage device 120, while VFs 325-1, 325-2, and 325-3 can be exposed by FPGA 315. Embodiments of the inventive concept operating using this implementation can avoid having FPGA 315 implement hardware PF functionality that has already been implemented by storage device 120, but supplement this hardware implementation with a number of VFs that may be greater than what storage device 120 itself can provide.

[0101] Regardless of whether storage device 120 or FPGA 315 implements PF 320, FPGA 315 can be inserted between processor 110 (and hypervisor 310) on one hand and storage device 120 on the other hand. Any communication between hypervisor 310 (and thus VMs 305-1, 305-2, and 305-3) and storage device 120 will pass through FPGA 315, thereby allowing FPGA 315 to enhance the functionality provided by storage device 120 (or potentially provide SR-IOV type support for storage device 120 when storage device 120 itself does not provide an SR-IOV type implementation).

[0102] In terms of implementation, FPGA 315 can be hardware within storage device 120 (i.e., storage device 120 can include FPGA 315 within its structure), or FPGA 315 can be additional hardware external to storage device 120, but still along the communication path between processor 110 and storage device 120. For example, FPGA 315 can be implemented as a circuit board installed in Figure 1 host device 105 that receives data from a peripheral component interconnect express (PCIe) bus via a connection from FPGA 315 to storage device 120. Regardless of how FPGA 315 is implemented, FPGA 315 should be somewhere between processor 110 and storage device 120 to capture information sent by hypervisor 310 and perform the functions of FPGA 315.

[0103] The hypervisor 310 can capture the control requests from VMs 305-1, 305-2, and 305-3 to manage their processing. For example, the hypervisor 310 can capture the information sent to the Peripheral Component Interconnect (PCI) configuration space or the requests sent to the control queue of the storage device 120, and process them locally, redirect the requests to the FPGA 315, or generate new requests similar to the original requests (although the specific manner depends on the particular request).

[0104] The FPGA 315 provides advantages over implementing the SR-IOV function within the storage device 120, where the FPGA 315 can be programmed in the field as part of the installation process: the storage device 120 is typically programmed during manufacturing. For example, the FPGA 315 may be able to support, say, 100 VFs of a storage device, but at installation, the customer may wish to expose only, say, 50 VFs (because Figure 1 the host device 105 may not be able to support that many VMs). The FPGA 315 can then be programmed using conventional techniques to expose only 50 VFs, leaving the remaining unused gates (and thus not consuming power). For example, the FPGA 315 can be programmed at manufacturing to have an enclosure that provides an interface using Non-Volatile Memory Express (NVMe) commands. After installing the FPGA 315 into Figure 1 the host device 105, the customer can use these NVMe commands to customize the FPGA 315 as desired.

[0105] The FPGA 315 can also be programmed as desired in terms of which parameters it can support. For example, as discussed below with reference to Fig. 9 , there are many different variations in QoS requirements. But the FPGA 315 can be programmed to consider only the bandwidth and ignore the parameters related to the number of I / O requests or the number of bytes processed per second. The same possibility holds for other parameters for the extended I / O queue creation command.

[0106] Figure 4 shows Figure 1 the communication path between the VMs of Figure 1 and the storage device 120 of Figure 1 , where the storage device 120 of Figure 4 exposes multiple physical functions. Figure 3 is the same as Figure 4 , except that instead of exposing one PF 320, three PFs 320, 405, and 410 are exposed in Figure 4FIG. 0 shows that the FPGA 315 exposes three PFs 320, 405, and 410, but embodiments of the inventive concept may support exposing any number of PFs, and for each PF, any number of VFs 325-1, 325-2, and 325-3 may be exposed. For many reasons, multiple PFs may be exposed: for example, the amount of storage provided by the storage device 120 may be too large for a single PF to fully support. Each PF typically requires some special hardware support. Thus, the more PFs to be exposed, the more hardware is generally required in the underlying device, increasing the chip size and power consumption and potentially degrading its performance as viewed by a single VM. Each PF may provide different functions, or multiple PFs may provide the same function (enabling a single device to support more than one VM at a time).

[0107] In Figure 3-Figure 4 , all functions providing SR-IOV type support described herein may be implemented in the FPGA 315, or some (or all) may be implemented in the storage device 120. For example, in some embodiments of the inventive concept, the storage device 120 may be a conventional storage device that does not provide built-in support for any SR-IOV type functions, where the FPGA 315 is responsible for all SR-IOV type functions. In other embodiments of the inventive concept, the storage device 120 may provide support for extended I / O queue creation commands (discussed below with reference to Figure 5-Figure 6 ), and if the storage device 120 includes sufficient I / O queues to support the expected number of VMs, the FPGA 315 may only provide the VFs 325-1, 325-2, and 325-3 and manage I / O requests from the VMs as a "pass-through" device. In yet another embodiment of the inventive concept, the storage device 120 may provide support for extended I / O queue creation commands, but because the number of VMs is expected to exceed the number of I / O queues of the storage device 120, the FPGA 315 may still manage I / O queue creation internally: then the storage device 120 may be regarded as a conventional storage device despite supporting extended I / O queue creation commands.

[0108] Figure 5 shows Figure 1 details of the storage device 120. In Figure 5 , the storage device 120 may include a host interface 505 that manages communication with the Figure 1 host device 105. The storage device 120 may also include a storage medium 510 that stores data used by the Figure 3The actual data accessed by VMs 305-1, 305-2, and 305-3. The memory 510 can take any desired form: for example, if the storage device 120 is a solid-state drive (SSD), the storage device 510 can take the form of flash memory, and if the storage device 120 is a hard disk drive, the storage device 510 can take the form of a disk. The storage device 120 can also include doorbell distribution logic 515, which can distribute doorbells across the internal memory of the storage device 120 such that each doorbell is located in a different page of the internal memory (where the size of the page can be determined according to Figure 1 the page size of the memory 130). By placing the doorbells in different pages, different VMs access different pages to access their (multiple) doorbells, thus avoiding the possibility that multiple VMs may need to access the same memory page to access their doorbells (which would defeat the goal of VM isolation). And although Figure 5 shows the doorbell distribution logic 515 as part of the storage device 120, embodiments of the inventive concept can also place the doorbell distribution logic 515 in Figure 3 the FPGA 315. Doorbell distribution will be further discussed below with reference to Figure 8 Further discussion will be provided below.

[0109] Figure 5 Other conventional hardware and / or circuitry that may be included in the storage device 120 are not shown, such as the Flash Translation Layer (FTL) and read / write circuitry for an SSD, or the read / write head for a hard disk drive. Figure 5 Additional optional components that may be included in the storage device 120 are also not shown, such as a cache. Embodiments of the inventive concept extend to include all such conventional components.

[0110] The storage device 120 can use any desired interface to communicate with Figure 1 the host device 105. For example, the storage device 120 can use an NVMe interface, or the storage device 120 can use a Serial ATA attachment interface. Similarly, the storage device 120 can use any desired connection mechanism to connect to the host device 105, including, for example, PCI or PCIe (using 4, 8, or any other number of PCI lanes), M.2, and SATA, among others. Embodiments of the inventive concept are intended to include all variations in connection and interface.

[0111] The storage device 120 can also include an I / O queue (also referred to as an I / O queue pair) that can be used for request submission and response return: a submission queue (SQ) can be Figure 3The VM 305-1 is used to submit I / O requests (such as read or write requests), and the completion queue (CQ) can be used by the storage device 120 to return the results to Figure 3 the VM 305-1. In Figure 5 it, the storage device 120 is shown as including three I / O queues 520-1, 520-2, and 520-3, having submission queues 525-1, 525-2, and 525-3 respectively and having completion queues 530-1, 530-2, and 530-3 respectively. Although Figure 5 it is proposed that each submission queue 520-1, 520-2, and 520-3 have a corresponding completion queue 525-1, 525-2, and 525-3, embodiments of the inventive concept can support a single completion queue that includes results from multiple submission queues.

[0112] The storage device 120 may include circuitry that supports I / O queue creation commands 535 that can be provided by the storage device 120. Using the I / O queue creation commands 535, Figure 3 the hypervisor 310 or Figure 3 the FPGA 315 can request the establishment of I / O queues for use. However, although conventional storage devices (such as those that support the NVMe interface) can provide I / O queue creation commands 535 in a standardized form, the illustrated I / O queue creation commands 535 represent extended I / O queue creation commands, thus providing additional attributes not provided or supported by conventional I / O queue creation commands.

[0113] Figure 6 is shown for Figure 1 the storage device 120 of Figure 5 the extended I / O queue creation command 535. In Figure 6 it, the I / O queue creation command 535 is shown. The I / O queue creation command 535 may include parameters that are part of the I / O queue creation command defined by the conventional NVMe specification, such as the queue size 603 (how much space should be allocated for the queue being established), the queue identifier 606 (the identifier of the submission queue being established), the completion queue identifier 609 (the identifier of the completion queue), the queue priority 612 (the relative priority of the submission queue being established), and the physical contiguity flag 615 (indicating whether the submission queue and the completion queue are physically contiguous). However, additionally, the I / O queue creation command 535 may include other attributes. These attributes include the logical block address (LBA) range attribute 618, the quality of service (QoS) attribute 621, and the shared namespace attribute 624. The LBA range attribute 618 can specify the range of LBAs associated with the VM 305-1. Using the LBA range attribute 618, Figure 3the FPGA 315 or Figure 1 the storage device 120 can allocate Figure 1 a portion of the physical block address (PBA) on the storage device 120 for use by that VM and not for use by other VMs (although exceptions to this rule are discussed below). By isolating the range of LBAs and the corresponding range of PBAs, VM isolation can be provided.

[0114] Figure 7 illustrates Figure 1 an example of how the physical storage of the storage device 120 is divided into multiple namespaces. In Figure 7 it, a 1 TB storage device 510 is illustrated Figure 5 although embodiments of the inventive concept can support storage devices having any total capacity. Figure 5 This 1 TB storage device 510 is shown as being divided into three namespaces 705-1, 705-2, and 705-3, but embodiments of the inventive concept can divide Figure 5 the storage device 510 into any number of namespaces.

[0115] Each namespace has an associated range of LBAs. Thus, namespace 705-1 includes the range 710-1 of LBAs, namespace 705-2 includes the range 710-2 of LBAs, and namespace 705-3 includes the range 710-3 of LBAs. Corresponding to each range 710-1, 710-2, and 710-3 of LBAs, ranges 715-1, 715-2, and 715-3 of PBAs can be established in Figure 5 the storage device 510. Each range 715-1, 715-2, and 715-3 of PBAs should be at least as large as the corresponding range of LBAs so that each LBA in a given LBA range has a corresponding PBA. Note that the LBAs and PBAs are not necessarily coincidental: for example, the range 710-2 of LBAs goes from starting LBA 0 to ending LBA 134,217,727 (which could be, for example, a block address where each block includes 4096 bytes of data: this block address corresponds to byte address 549,755,813,887), while the range 715-2 of PBAs goes from starting PBA 67,108,864 (byte address 274,877,906,944) to ending PBA 201,326,591 (byte address 824,633,720,831).

[0116] One advantage of mapping LBA ranges to PBAs and associating one (or both) of these ranges with an I / O queue is that such mapping can avoid the mixer effect. The mixer effect is a result of how traditional storage devices handle I / O requests. While an I / O request is still within the I / O queue, the I / O request has some residual context. However, once the I / O request reaches the FTL, any such context is lost: all I / O requests look the same at that point. As a result, Figure 1 storage device 120 will not be able to guarantee the QoS requirements of a particular VM. However, if the PBA can be tied back to a particular I / O queue via the LBA-PBA mapping (which is itself reversible), then the QoS requirements of that I / O queue can still be Figure 1 located and met by storage device 120.

[0117] It is noted that each namespace 705-1, 705-2, and 705-3 has its own namespace identifier (ID). In some embodiments of the inventive concept, these namespace IDs may correspond to the queue identifiers provided as Figure 6 queue identifier 606.

[0118] Return Figure 6 , the LBA range attribute 618 can be represented in a variety of ways. For example, the LBA range attribute 618 can include an LBA start 627 and an LBA end 630, which provide the start and end addresses of the LBAs within the Figure 7 LBA ranges 710-1, 710-2, and 710-3. Alternatively, given the LBA start 627 and the queue size 603, it is possible to infer the LBA end 630, which would permit the omission of the LBA end 630. The structure of the extended I / O queue creation command 535 can be established to support any desired variations on the LBA range attribute 618, including and / or omitting parameters as needed.

[0119] The QoS attribute 621 can represent any desired QoS specification. For example, the queue being established by the I / O queue creation command 535 can be associated with a VM having a service level agreement (SLA) that attempts to guarantee a particular service level for the VM user. The QoS attribute 621 can be represented in any desired form. Figure 6Some example forms shown include minimum guaranteed bandwidth 633, maximum guaranteed bandwidth 636, minimum number of read requests per second 639, maximum number of read requests per second 642, minimum number of read bytes per second 645, maximum number of read bytes per second 648, minimum number of write requests per second 651, maximum number of write requests per second 654, minimum number of write bytes per second 657, and maximum number of write bytes per second 660. Note that traditional SR-IOV solutions rely on Figure 3 the hypervisor 310 to manage QoS requirements; embodiments of the inventive concept can shift such management to Figure 3 the FPGA 315 or Figure 1 the storage device 120, thereby reducing Figure 3 the load on the hypervisor 310 and the host CPU load, and thereby improving overall system performance. The structure of the extended I / O queue creation command 535 can be established to support any desired changes to the QoS attributes 621, including and / or omitting parameters as needed.

[0120] Finally, the shared namespace attribute 624 can represent a list of namespaces intended to be shared by VMs requesting to create I / O queues for access to the physical storage device. The concept of the shared namespace attribute represents an exception to the concept of VM isolation: if virtual machines share access to a common data set, they may not be isolated. However, since there may be situations where multiple VMs need to share information, and it is more efficient for them to share access to a common data set than access to message data between VMs, the shared namespace attribute 624 provides such a solution. The shared namespace attribute 624 can be implemented in any desired manner: one example is that the I / O queue creation command 535 includes an array 663 of shared namespace IDs that lists the namespace IDs of the VMs that are to have shared access to the data set.

[0121] Figure 8 Shows Figure 1 the memory mapping of the doorbell in the storage device 120 to support VM isolation. In traditional storage devices that use a doorbell to communicate between Figure 1 the host device 105 and Figure 1 the storage device 120, there can be multiple doorbells: for example, Figure 5 each of the I / O queues 520-1, 520-2, and 520-3 has a doorbell. To support access to these doorbells by VMs 305-1, 305-2, and 305-3, Figure 1 the storage device 120 can request that a portion of the address space of Figure 1 the host device 105 be allocated to the storage device 120. Then, Figure 1Addresses in the address space of host device 105 can be mapped to memory addresses of storage device 105. Then, Figure 3 hypervisor 310 can provide the addresses of the doorbells to Figure 3 VMs 305-1, 305-2, and 305-3 such that Figure 3 VMs 305-1, 305-2, and 305-3 can access the doorbells by using Figure 1 addresses in the address space of host device 105.

[0122] However, in Figure 1 a conventional implementation of storage device 120, these doorbells can reside in contiguous segments of memory: Figure 8 This is illustrated. In Figure 8 , memory 805 is shown, which represents Figure 1 the memory within storage device 120 or Figure 3 the memory within FPGA 315. Figure 5 Memory addresses 810-1, 810-2, and 810-3 of the doorbells for I / O queues 520-1, 520-2, and 520-3 are shown to occupy contiguous segments of memory and all be within a single page of memory 805. Although Figure 8 three doorbells are shown occupying memory addresses 810-1, 810-2, and 810-3, embodiments of the inventive concept can include any number of doorbells. In Figure 1 the case where host device 105 operates as a single device (without virtualization), this arrangement works well. However, in Figure 1 the case where host device 105 supports multiple virtual machines that require isolation, having all doorbells reside in the same page of memory 805 means that the VMs must share access to the same page of memory, which violates the requirements of VM isolation.

[0123] To address this difficulty, Figure 1 storage device 120 or Figure 3 FPGA 315, depending on which device provides the doorbells, can locate the doorbell memory addresses in different pages of Figure 1 memory 130, as shown in memory 815. Figure 1 storage device 120 or Figure 3An example of how the FPGA 315 locates the doorbell memory address in different pages is shown in U.S. Patent Application Serial No. 14 / 862,145, filed on September 22, 2015, which is now pending and has been published as U.S. Patent Publication No. 2016 / 0306580, incorporated herein by reference. By using a doorbell stride value, the doorbell memory address can be shifted such that each doorbell is located Figure 1 in the storage device 120 of Figure 3 or in different pages of the FPGA 315 of Figure 1 (in this context, the term "different pages" is intended to describe intervals such that Figure 1 the corresponding doorbell addresses in the address space of the host device 105 of Figure 3 are in different pages based on the size of the pages in the O / S memory). Then, because the doorbells are in different pages of Figure 1 the storage device 120 of Figure 1 or in different pages of the FPGA 315 of

[0124] in Figure 5-Figure 7 the corresponding addresses in the address space of the host device 105 of Figure 5 are also in different pages of the O / S memory. Thus, for example, memory address 810-1 can be mapped to memory address 820-1, memory address 810-2 can be mapped to memory address 820-2, memory address 810-3 can be mapped to memory address 820-3, and so on. Since pages 825-1, 825-2, 825-3, and 825-4 can represent different pages of the memory, after this mapping, each doorbell can reside in a different page of the memory 815. As a result, the VMs do not share access to a single page of Figure 1 the memory 130 to access the doorbells, thus supporting VM isolation. Figure 1 the storage device 120 of Figure 1 the storage device 120 of Figure 3 While the storage device 120 of Figure 1 the storage device 120 of Figure 1 the storage device 120 of Figure 1 the storage device 120 of Figure 3 the FPGA 315 in the system can enhance Figure 1 Function of storage device 120

[0125] First, the FPGA 315 may include the necessary hardware to support VM isolation. Referring back Figure 3 , since each request to be sent to Figure 1 the storage device 120 may pass through Figure 3 the FPGA 315, the FPGA 315 may intercept requests intended for Figure 1 the storage device 120 to receive. For example, if Figure 1 the storage device 120 does not support Figure 5 the extended I / O queue creation command 535, then Figure 3 the FPGA 315 may intercept any such requests and handle I / O queue creation itself. Figure 3 The FPGA 315 may send a traditional I / O queue creation command to Figure 1 the storage device 120 while managing VM isolation itself, thus providing QoS guarantees and sharing namespaces when appropriate. To this end, Figure 3 the FPGA 315 may include hardware similar to what may otherwise be included in Figure 1 the storage device 120. Figure 3 The FPGA 315 may determine whether a particular I / O request in an I / O queue is within the range of the LBA 710 associated with that I / O queue (or part of a shared namespace). Figure 3 The FPGA 315 may organize and sort I / O requests to be sent to Figure 1 the storage device 120 in order to provide QoS guarantees.

[0126] To support I / O queue creation and VM isolation, Figure 3 the FPGA 315 may include a virtual I / O queue creation command 903, as Fig. 9 shown. The virtual I / O queue creation command 903 is very similar to Figure 6 the I / O queue creation command 535, and includes similar parameters and attributes. The main difference is that, given that Figure 6 the I / O queue creation command 535 is intended to be processed by Figure 1 the storage device 120 (although as described above, Figure 3 the FPGA 315 may intercept Figure 6 the I / O queue creation command 535 and process it internally instead), while the virtual I / O queue creation command 903 points to Figure 3 the FPGA 315, rather than being intended to be processed by Figure 1 the storage device 120.

[0127] Because the virtual I / O queue creation command 903 is intended to achieve results similar to those of the I / O queue creation command 535, the virtual I / O queue creation command 903 can include attributes / parameters similar to those of the I / O queue creation command 535 of Fig. 9 Thus, the virtual I / O queue creation command can include parameters that are part of a traditional I / O queue creation command, such as the queue size 906 (how much space should be allocated for the queue being established), the queue identifier 909 (the identifier of the submission queue being established), the completion queue identifier 912 (the identifier of the completion queue), the queue priority 915 (the relative priority of the submission queue being established), and the physical contiguity flag 918 (indicating whether the submission queue and the completion queue are physically contiguous). The virtual I / O queue creation command 903 can also include extended attributes, such as the LBA range attribute 921, the QoS attribute 924, and the shared namespace attribute 927.

[0128] Just as Figure 5 the I / O queue creation command 535, the LBA range attribute 921 can be represented in a variety of ways. For example, the LBA range attribute 921 can include the LBA start 930 and the LBA end 933, which provide the start and end addresses of the LBAs within the LBA ranges 710-1, 710-2, and 710-3 of Figure 7 Alternatively, given the LBA start 930 and the queue size 906, it is possible to infer the LBA end 933, which would permit the omission of the LBA end 933. The structure of the extended virtual I / O queue creation command 903 can be established to support any desired variations on the LBA range attribute 921, including and / or omitting parameters as needed.

[0129] Similarly, the QoS attribute 924 can represent any desired QoS provisions. For example, the queue being established by the virtual I / O queue creation command 903 can be associated with a VM that has an SLA that attempts to guarantee a particular service level for the VM user. The QoS attribute 924 can be represented in any desired form. Fig. 9 Some example forms shown include the minimum guaranteed bandwidth 936, the maximum guaranteed bandwidth 939, the minimum number of read requests per second 942, the maximum number of read requests per second 945, the minimum number of bytes read per second 948, the maximum number of bytes read per second 951, the minimum number of write requests per second 954, the maximum number of write requests per second 957, the minimum number of bytes written per second 960, and the maximum number of bytes written per second 963. The structure of the extended virtual I / O queue creation command 903 can be established to support any desired variations on the QoS attribute 924, including and / or omitting parameters as needed.

[0130] Finally, the shared namespace attribute 927 can represent a list of namespaces that are intended to request that VMs sharing access to a physical storage device create I / O queues. The concept of the shared namespace attribute represents an exception to the concept of VM isolation: If virtual machines share access to a common data set, they may not be isolated. However, since there may be cases where multiple VMs need to share information, and it is more efficient for them to share access to a common data set than access to message data between VMs, the shared namespace attribute 927 provides such a solution. The shared namespace attribute 927 can be implemented in any desired manner: An example is that the virtual I / O queue creation command 903 includes an array 966 of shared namespace IDs that lists the namespace IDs of the VMs that are to share access to the data set.

[0131] Note that, in some embodiments of the inventive concept, Figure 1 the storage device 120 of Figure 6 can include the I / O queue creation command 535 of Figure 3 and thus can support creating an I / O queue and allocating it to VMs 305-1, 305-2, and 305-3. In such embodiments of the inventive concept, Figure 5 the FPGA 315 of Figure 1 can simply accept the virtual I / O queue creation command 903 and send it as Figure 3 the I / O queue creation command 535 of

[0132] to the storage device 120 of Figure 3 instead of processing the virtual I / O queue creation command 903 in the FPGA 315 of Figure 3 to create a virtual I / O queue. Figure 3 However, the FPGA 315 of Figure 1 can do more than offload the hardware that supports SR-IOV type operations (enabling the FPGA 315 of Figure 7 to be used with a storage device that does not itself include SR-IOV type hardware). Figure 1 The FPGA 315 of Figure 1 can also extend the number of I / O queues "provided" by the storage device 120 of Figure 1 Since each VM on the host device 105 of Figure 1 needs its own access to the storage device 120 of Figure 1The upper bound on the number of VMs of the storage device 120. This upper bound may be much lower than Figure 3 the hypervisor 310 (and Figure 1 the processor 110) can support, which means that either the VMs will not have access to Figure 1 the storage device 120, or Figure 1 the resources of the host device 105 will be idle.

[0133] Figure 3 The FPGA 315 can solve this problem by providing additional virtual I / O queues, and the number of such virtual I / O queues may be much higher than the number of I / O queues (and PF / VFs) directly provided by Figure 1 the storage device 120. Fig.10 Illustrates this situation.

[0134] In Fig.10 , it is assumed that the I / O queues 520-1, 520-2, and 520-3 represent the only I / O queues provided by Figure 1 the storage device 120 (in fact, the number of I / O queues is greater than three, but generally still less than Figure 1 the number of VMs that the host device 105 can support simultaneously). If Figure 1 the host device 105 is running more than three VMs, then if Figure 1 the storage device 120 is the only source of I / O queues, one or more VMs will have no available I / O queues. However, in the case where the FPGA 315 supports virtual I / O queues, these additional VMs can still access Figure 1 the storage device 120.

[0135] When Figure 3 the hypervisor 310 issues a virtual I / O queue creation command 903 provided by the FPGA 315, the FPGA 315 can establish a new virtual I / O queue. Fig.10 Illustrates five virtual I / O queues 1005-1, 1005-2, 1005-3, 1005-4, and 1005-5, but the FPGA 315 can support any number of virtual I / O queues (the limit is defined by the number of gates in the FPGA 315). Each virtual I / O queue includes its own submission queue and completion queue. Thus, the virtual I / O queues 1005-1, 1005-2, 1005-3, 1005-4, and 1005-5 include submission queues 1010-1, 1010-2, 1010-3, 1010-4, and 1010-5 respectively, and include completion queues 1015-1, 1015-2, 1015-3, 1015-4, and 1015-5 respectively.

[0136] Each virtual I / O queue can then be associated with Figure 1 the (hardware) I / O queues of the storage device 120. The FPGA 315 can use mapping logic 1020, which can organize the virtual I / O queues 1005-1 to 1005-5 into groups using any desired method and map them to the (hardware) I / O queues 520-1, 520-2, and 520-3. For example, in Fig.10 , the mapping logic 1020 can select virtual I / O queues 1005-1 and 1005-2 to form group 1025-1 associated with I / O queue 520-1, select virtual I / O queue 1005-3 (by itself) to form group 1025-2 associated with I / O queue 520-2, and select virtual I / O queues 1005-4 and 1005-5 to form group 1025-3 associated with I / O queue 520-3. Thus, any I / O requests received by the FPGA 315 from Figure 3 the VMs 305-1, 305-2, and 305-3 can be "placed" in the appropriate virtual submission queue and then passed to Figure 1 the correct (hardware) submission queue of the storage device 120. Similarly, responses received from the (hardware) completion queue can be "placed" in the correct virtual completion queue and "returned" to the appropriate VM.

[0137] The mapping logic 1020 can organize the virtual I / O queues 1005-1 to 1005-5 into groups using any desired method, where each group is associated with a particular (hardware) I / O queue. For example, the mapping logic 1020 can randomly assign the virtual I / O queues to groups. Alternatively, the mapping logic 1020 can assign the virtual I / O queues to groups in a round-robin manner: the first virtual I / O queue is assigned to the first group, the second virtual I / O queue is assigned to the second group, and so on until all groups have a virtual I / O queue, after which, starting from the first group again, the virtual I / O queues are assigned to groups. Alternatively, the mapping logic 1020 can assign the virtual I / O queues to groups based on Figure 3 the expected I / O load of the VMs 305-1, 305-2, and 305-3 to attempt to balance the I / O load between groups. Alternatively, the mapping logic 1020 can assign the virtual I / O queues to groups based on the relative priorities specified for Figure 3 the VMs 305-1, 305-2, and 305-3 (note that Figure 6 the queue priority 612 is part of the I / O queue creation command defined by the traditional NVMe specification, and Fig. 9 the queue priority 915 is part of Fig. 9 the virtual I / O queue creation command 903). Alternatively, the mapping logic 1020 can be based on Fig. 9 The QoS attribute 924 of Fig. 9 assigns virtual I / O queues to groups in an attempt to meet the QoS requirements of VMs 305-1, 305-2, and 305-3. Embodiments of the inventive concept may also employ other methods to determine which virtual I / O queues to assign to which groups.

[0138] Since the FPGA 315 knows which LBAs are associated with each virtual I / O queue (via Fig. 9 the LBA range attribute 921 of Fig. 9 and / or Fig. 9 the shared namespace attribute 927 of Fig. 9 ), the FPGA 315 can enforce VM isolation by rejecting I / O requests that are not suitable for that virtual I / O queue. Similarly, because the FPGA 315 knows the QoS requirements of the VMs (via Fig. 9 the QoS attribute 924 of Fig. 9 ), the FPGA 315 can forward I / O requests to I / O queues 520-1, 520-2, and 520-3 in a manner that meets the QoS requirements of each VM. Thus, for example, if a VM has established a QoS requirement of at least 10 I / O requests per second (assuming that there may be I / O requests pending), the FPGA 315 can prioritize the I / O requests from the corresponding virtual I / O queue before forwarding I / O requests from other virtual I / O queues. (This example is somewhat arbitrary as it implies that no other VMs have QoS requirements: in the case where multiple VMs have QoS requirements, the FPGA 315 can manage the I / O requests in a manner that meets the QoS requirements of all VMs). Other QoS requirements, such as bandwidth requirements, can be similarly handled via the FPGA 315.

[0139] Fig.11 illustrates an example process flow diagram of Figure 1 the storage device 120 of Figure 1 allocating Figure 3 the I / O queues 520 of Figure 3 to Figure 5 the VMs 305 of Figure 5 according to an embodiment of the inventive concept. In Fig.11 , at block 1105, Figure 1 the storage device 120 of Figure 1 can receive Figure 3 the I / O queue creation command 535 of Figure 3 from Figure 5 the hypervisor 310 of Figure 5 . Alternatively, at block 1110, Figure 1 the storage device 120 of Figure 1 can receive Figure 3 the I / O queue creation command 535 of Figure 3 from Figure 5 the FPGA 315 of Figure 5 . In either case, Figure 5 the I / O queue creation command 535 of Figure 5 can include Figure 7 the range 710 of LBAs of Figure 7 . Regardless of Figure 5Regardless of the source of the I / O queue creation command 535, at block 1115, Figure 1 the storage device 120 can establish Figure 5 the I / O queue 520. At block 1120, Figure 1 the storage device 120 can select a range 715 of PBAs large enough to support the received Figure 7 range 710 of LBAs. At block 1125, Figure 7 the storage device 120 can map Figure 1 the range 710 of LBAs to Figure 7 the range 715 of PBAs. Thus, when Figure 7 the storage device 120 receives an I / O request, it can access only the range 715 of PBAs corresponding to Figure 1 the VM 305, thereby isolating each VM from other VMs. In a similar manner, since Figure 3 the mapping between the range 710 of LBAs and Figure 7 the range 715 of PBAs is invertible, given a particular physical address, the appropriate context of the I / O request can be determined such that an appropriate completion queue can be selected to notify Figure 7 the VM 305 that the I / O request has been completed, as Figure 7 suggested in block 1130 (where the success of the I / O request can be returned to Figure 3 the VM 305). Fig.11 Figure 3 the VM 305).

[0140] Note that blocks 1105 and 1110 suggest using Figure 5 the extended I / O queue creation command 535, regardless of Figure 5 the source of the I / O queue creation command 535. While embodiments of the inventive concept include such a possibility, other embodiments of the inventive concept may include Figure 1 a storage device 120 that does not support Figure 5 the extended I / O queue creation command 535. In these embodiments of the inventive concept, Figure 3 the FPGA 315 can support simulating all functions of SR-IOV with Figure 1 the storage device 120, and Figure 1 the storage device 120 processes I / O requests without any context of the I / O request. In other words, Figure 3 the FPGA 315 can support all context management and VM isolation, enabling Figure 1 the storage device 120 to operate as a traditional storage device without implementing any functions of the inventive concept.

[0141] ​ Fig.12 Shows an Figure 3 FPGA 315 of Figure 3 allocates a Fig.10 virtual I / O queue 1005 of Fig.12 . In Figure 3 , the FPGA 315 of Figure 3 can receive a Fig.10 virtual I / O queue creation command 903 from the Figure 3 hypervisor 310 of Figure 3 . At block 1210, the Fig.10 FPGA 315 of Figure 3 can establish a Figure 1 virtual I / O queue 1005 for the Figure 5 VM 305 of Figure 1 . At block 1215, the Figure 1 FPGA 315 of Figure 5 can send an Figure 3 I / O queue creation command 535 to the Figure 1 storage device 120 of Figure 5 . Note that if the

[0142] storage device 120 of Fig.10 supports it, the Fig.10 I / O queue creation command 535 sent to the Figure 1 storage device 120 of Figure 5 can be an extended version of the command, or if not supported, it can be a traditional I / O queue creation command. At block 1220, the Figure 3 FPGA 315 of Figure 7 can receive the Fig.10 result of the Figure 3 I / O queue creation command 535 from the Figure 3 storage device 120 of

[0143] Fig.12 The above several notes are in order. First, note that although in Fig.11 blocks 1105 and 1110 of Figure 1 , the Figure 5The extended I / O queue creation command 535 (in Figure 3 The FPGA 315 implements all functions of the present invention and Figure 1 The storage device 120 is a conventional storage device in the embodiment of the present invention concept), but Figure 3 This is not the case with the FPGA 315. Figure 3 The management program 310 sends to Figure 1 Any command from the storage device 120, or from Figure 3 VM 305 is sent to Figure 1 Any I / O request to the storage device 120 goes through Figure 3 FPGA 315. Since such a command will include an I / O queue creation command (either real or virtual), it can be expected that Figure 3 FPGA 315 receives the extended I / O queue creation command. If Figure 1 The storage device 120 may implement Figure 5 The I / O queue creation command 535 is Figure 3 The FPGA 315 does not need to receive Fig.10 Virtual I / O queue creation command 903 (although embodiments of the inventive concept may include Figure 3 The management program 310 Figure 3 FPGA 315 sends Fig.10 The virtual I / O queue creation command 903 is left to Figure 3 FPGA 315 to execute commands itself or to Figure 1 The storage device 120 issues Figure 5 I / O queue creation command 535). And if Figure 3 The FPGA 315 performs queue-to-VM allocation and context management, then Figure 3 The FPGA 315 will wish to receive Fig. 9 Virtual I / O queue creation command 903.

[0144] Second, note that blocks 1215 and 1220 assume that Figure 5 The I / O queue 520 has not been established on the storage device 120. If during execution, Figure 5 I / O queue 520 is already in Figure 1 120 (for example, if Figure 3 The FPGA315 is now Fig.10 The multiple virtual I / O queues 1005 are mapped to Figure 5 The storage device 120 Figure 5 If a separate I / O queue 520 is provided, blocks 1215 and 1220 may be omitted, as indicated by dashed line 1240.

[0145] Third, Figure 3 the FPGA 315 of Fig.10 may not need to store the context information of the virtual I / O queue 1005 of Figure 1 the storage device 120 of Figure 7 maps the range 710 of the LBA of Figure 7 to the range 715 of the PBA of Figure 5 and stores the context information of the Figure 3 VM 305 of the I / O queue 520 of

[0146] Fig.13 shows an example process flow diagram of the Figure 3 hypervisor 310 of Figure 3 processing control requests from the Fig.13 VM305-1, 305-2, and 305-3 according to an embodiment of the inventive concept. In Figure 3 the hypervisor 310 of Figure 3 may receive a control request from the Figure 3 VM 305 of Figure 3 At block 1305. At block 1310, Figure 3 the hypervisor 310 of Figure 3 may capture the request. Then, at block 1315, Figure 3 the hypervisor 310 of Figure 3 may send a different (if similar) request to the Figure 3 FPGA 315 of Figure 1 : the second request may simulate the original request. At block 1320, Figure 3 the hypervisor 310 of Figure 3 may receive the result from the Figure 3 FPGA 315 of Figure 3 Note that the

[0147] Fig.14 shows a method for Figure 1 the storage device 120 of Figure 3 or the Figure 8 FPGA 315 of Figure 8Flowchart of an example process for different operating system pages 825 to support VM isolation. Note that the example process is the same whether implemented by the Figure 1 storage device 120 of Figure 3 or the FPGA 315 of Figure 3 . For descriptive purposes, the FPGA 315 of Figure 1 will be described as performing the example process, but embodiments of the inventive concept extend to the storage device 120 of

[0148] that also performs the example process. Fig.14 In Figure 3 , at block 1405, the FPGA 315 of Figure 3 can identify a doorbell for managing communication between the FPGA 315 of Figure 3 and the VM 305 of Figure 3 . At block 1410, the FPGA 315 of Figure 1 can distribute the memory addresses 820 of Figure 1 on different memory pages based on the page size of the memory 130 in the host device 105 of Figure 8 . For example, the FPGA 315 of Figure 3 can use the doorbell span value to locate the doorbell memory addresses 820 of Figure 8 in different pages. At block 1415, the FPGA 315 of Figure 3 can request an address space from the host device 105 of Figure 1 . At block 1420, the FPGA 315 can map the memory addresses 820 of Figure 8 to memory addresses in the requested address space. In this way, VM isolation can be maintained because no two VMs 305 can access their doorbells on a common memory page. At block 1425, the FPGA 315 of Figure 3 can provide the new memory addresses 820 of Figure 3 to the VM 305 of Figure 8 .

[0149] At block 1430, the FPGA 315 of Figure 3 can receive a request from the VM 305 of Figure 3 to access the doorbell at the mapped memory address. At block 1435, the FPGA 315 of Figure 3 can reverse the mapping and restore the original memory addresses 820 of Figure 8 . At block 1440, the FPGA 315 of Figure 3 can send the request to the original memory addresses 820 of Figure 8 .

[0150] In Figure 11-Figure 14 are shown some embodiments of the inventive concept. However, those skilled in the art will recognize that other embodiments of the inventive concept are possible by changing the order of blocks, omitting blocks, or by including blocks not shown in the figures. All such variations of the flowcharts are considered embodiments of the inventive concept, whether explicitly described or not.

[0151] Embodiments of the inventive concept provide several technical advantages over the prior art. First, by removing the hardware requirements in the storage device 120 of Figure 1 to support VM isolation and using the FPGA 315 of Figure 3 , in theory any storage device can be used in a system that requires SR-IOV type functionality because Figure 3 the FPGA 315 of Figure 3 can enforce VM isolation. Second, since the FPGA 315 of Figure 3 can be programmed at installation, the specific desired functionality to be provided by the system can be established at installation rather than being fixed at the manufacturing point of the storage device 120 (which may not provide the best solution for all installations). Third, Figure 3 the FPGA 315 of Figure 1 can provide more VFs than the storage device 120 of Figure 1 , enabling the use of the storage device 120 of Figure 1 in a system with a larger number of VMs. Fourth, although VM isolation is provided by the FPGA 315 of Figure 1 Figure 3 , for I / O requests, each VM can still have "bare metal" access to the storage device 120 of Figure 1 . Fifth, the doorbell memory address can be remapped to different O / S memory pages, further enhancing VM isolation. And sixth, because the context of an I / O request can be traced to a particular I / O queue, the storage device 120 of Figure 1 can still support the QoS requirements of the VMs even after being removed from that I / O queue, thus avoiding the mixer effect.

[0152] The following discussion is intended to provide a brief, general description of one or more suitable machines in which aspects of the inventive concept may be implemented. One or more machines may be controlled, at least in part, by input from conventional input devices such as keyboards, mice, etc., as well as by instructions received from another machine, interaction with a virtual reality (VR) environment, biofeedback, or other input signals. As used herein, the term "machine" is intended to broadly encompass a single machine, virtual machine, or system of communicatively coupled machines, virtual machines, or devices operating together. Exemplary machines include computing devices such as personal computers, workstations, servers, portable computers, handheld devices, telephones, tablets, etc., and transportation devices such as private or public transportation, e.g., cars, trains, taxis, etc.

[0153] One or more machines may include an embedded controller such as a programmable or non-programmable logic device or array, an application specific integrated circuit (ASIC), an embedded computer, a smart card, etc. One or more machines may utilize one or more connections to one or more remote machines, such as via a network interface, modem, or other communicative coupling. Machines may be interconnected via physical and / or logical networks such as intranets, the Internet, local area networks, wide area networks, etc. Those skilled in the art will understand that network communication may utilize a variety of wired and / or wireless short-range or long-range carriers and protocols, including radio frequency (RF), satellite, microwave, Institute of Electrical and Electronics Engineers (IEEE) 802.11, optical, infrared, cable, laser, etc.

[0154] Embodiments of the inventive concept may be described by reference to or in conjunction with associated data, including functions, procedures, data structures, applications, etc., which, when accessed by a machine, cause the machine to perform tasks or define abstract data types or low-level hardware contexts. The associated data may be stored, for example, in volatile and / or non-volatile memory (e.g., RAM, ROM, etc.), or other storage devices and their associated storage media (including hard disk drives, floppy disks, optical storage devices, magnetic tapes, flash memories, memory sticks, digital video disks, biological storage devices, etc.). The associated data may be transmitted in the form of packets, serial data, parallel data, propagated signals, etc. over a transmission environment including physical and / or logical networks. And may be used in compressed or encrypted form. The associated data may be used in a distributed environment and stored locally and / or remotely for access by machines.

[0155] Embodiments of the inventive concept may include a tangible, non-transitory machine-readable medium including instructions executable by one or more processors, the instructions including instructions for performing the elements of the inventive concept as described herein.

[0156] The various operations of the above method can be performed by any suitable means capable of performing these operations, such as various (multiple) hardware and / or software components, circuits, and / or (multiple) modules. The software may include an ordered list of executable instructions for implementing logical functions and may be embodied in any “processor-readable medium” for use by or in connection with an instruction execution system, apparatus, or device, such as a single-core or multi-core processor or a system including a processor.

[0157] The blocks or steps of the methods or algorithms and functions described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. If implemented in software, these functions may be stored on or transmitted over a tangible, non-transitory computer-readable medium as one or more instructions or code. The software modules may reside in Random Access Memory (RAM), flash memory, Read Only Memory (ROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), registers, hard disk, removable disk, CD ROM, or any other form of storage medium known in the art.

[0158] The principles of the inventive concept have been described and illustrated with reference to the illustrated embodiments. It should be recognized that the illustrated embodiments may be modified in arrangement and detail without departing from these principles and may be combined in any desired manner. And, although the foregoing discussion has focused on particular embodiments, other configurations may also be considered. In particular, even when expressions such as “embodiments according to the inventive concept” are used herein, these phrases are meant to generally refer to the possibility of embodiments and do not mean to limit the inventive concept to a particular embodiment configuration. As used herein, these terms may refer to the same or different embodiments that may be combined into other embodiments.

[0159] The foregoing illustrative embodiments should not be construed as limiting the inventive concept thereof. Although some embodiments have been described, those skilled in the art will readily understand that many modifications can be made to these embodiments without materially departing from the novel teachings and advantages of the present disclosure. Accordingly, all such modifications are intended to be included within the scope of the inventive concept defined in the claims.

[0160] Embodiments of the inventive concept may be extended to the following statements, but are not limited thereto:

[0161] Statement 1. An embodiment of the inventive concept includes a storage device, comprising:

[0162] a storage means for data; and

[0163] at least one input / output (I / O) queue for requests from at least one virtual machine (VM) on a host device,

[0164] wherein the storage device supports an I / O queue creation command to request an I / O queue for allocating at least one I / O queue to a VM among at least one VM, the I / O queue creation command including an LBA range attribute of a range of logical block addresses (LBAs) to be associated with the I / O queue, and

[0165] wherein the storage device maps the range of LBAs to a range of physical block addresses (PBAs) in the storage means.

[0166] Statement 2. An embodiment of the inventive concept includes the storage device according to Statement 1, wherein:

[0167] the storage device includes a solid state drive (SSD) storage device; and

[0168] the SSD storage device uses a non-volatile memory express (NVMe) interface to the host device.

[0169] Statement 3. An embodiment of the inventive concept includes the storage device according to Statement 2, wherein the I / O queue includes a submission queue and a completion queue.

[0170] Statement 4. An embodiment of the inventive concept includes the storage device according to Statement 2, wherein the storage device receives the I / O queue creation command from a hypervisor on the host device.

[0171] Statement 5. An embodiment of the inventive concept includes the storage device according to Statement 2, wherein the LBA range attribute includes a start LBA and an end LBA.

[0172] Statement 6. An embodiment of the inventive concept includes the storage device according to Statement 2, wherein:

[0173] The LBA range attribute includes a start LBA; and

[0174] The I / O queue creation command further includes a queue size.

[0175] Claim 7. An embodiment of the inventive concept includes a storage device according to Claim 2, wherein the I / O queue creation command includes a quality of service (QoS) attribute for a QoS parameter for a VM.

[0176] Claim 8. An embodiment of the inventive concept includes a storage device according to Claim 7, wherein the QoS attribute is extracted from a set including a minimum bandwidth, a maximum bandwidth, a minimum number of read requests per second, a maximum number of read requests per second, a minimum number of bytes read per second, a maximum number of bytes read per second, a minimum number of write requests per second, a maximum number of write requests per second, a minimum number of bytes written per second, and a maximum number of bytes written per second.

[0177] Claim 9. An embodiment of the inventive concept includes a storage device according to Claim 2, wherein the I / O queue creation command includes a shared namespace attribute that specifies an array of namespaces that share access to a range of LBAs.

[0178] Claim 10. An embodiment of the inventive concept includes a storage device according to Claim 2, further including a doorbell distribution logic for locating a first doorbell of an I / O queue in a memory page different from a second doorbell of a second I / O queue.

[0179] Claim 11. An embodiment of the inventive concept includes a storage device according to Claim 2, further including a field programmable gate array (FPGA) that maps a plurality of virtual I / O queues to the I / O queue, wherein:

[0180] The FPGA supports a virtual I / O queue creation command to request allocation of a first virtual I / O queue of a plurality of virtual I / O queues, the virtual I / O queue creation command including a second LBA range attribute of a second range of logical block addresses (LBAs) to be associated with the first virtual I / O queue, the first virtual I / O queue being associated with the I / O queue.

[0181] Claim 12. An embodiment of the inventive concept includes a storage device according to Claim 11, wherein the FPGA includes a mapping logic for mapping the first virtual I / O queue to the I / O queue on the storage device such that an I / O request received from a VM in the first virtual I / O queue is passed to the storage device via the I / O queue, and a result received from the storage device in the I / O queue is passed to the VM via the first virtual I / O queue.

[0182] Claim 13. An embodiment of the inventive concept includes the storage device according to Claim 11, wherein a second virtual I / O queue of the FPGA is associated with the I / O queue.

[0183] Claim 14. An embodiment of the inventive concept includes the storage device according to Claim 11, wherein the FPGA is operable to invoke an I / O queue creation command on the storage device.

[0184] Claim 15. An embodiment of the inventive concept includes the storage device according to Claim 11, wherein the FPGA is operable to receive a virtual I / O queue creation command from a hypervisor on a host device.

[0185] Claim 16. An embodiment of the inventive concept includes the storage device according to Claim 15, wherein the FPGA is operable to receive a virtual I / O queue creation command from a hypervisor of a VM on a host device.

[0186] Claim 17. An embodiment of the inventive concept includes the storage device according to Claim 11, wherein the second LBA range attribute includes a second start LBA and a second end LBA.

[0187] Claim 18. An embodiment of the inventive concept includes the storage device according to Claim 11, wherein:

[0188] the second LBA range attribute includes a second start LBA; and

[0189] the virtual I / O queue creation command further includes a second queue size.

[0190] Claim 19. An embodiment of the inventive concept includes the storage device according to Claim 11, wherein the virtual I / O queue creation command includes a second QoS attribute for quality of service parameters of the VM.

[0191] Claim 20. An embodiment of the inventive concept includes the storage device according to Claim 19, wherein the second QoS attribute is extracted from a set including a second minimum bandwidth, a second maximum bandwidth, a second minimum number of read requests per second, a second maximum number of read requests per second, a second minimum number of read bytes per second, a second maximum number of read bytes per second, a second minimum number of write requests per second, a second maximum number of write requests per second, a second minimum number of write bytes per second, and a second maximum number of write bytes per second.

[0192] Claim 21. An embodiment of the inventive concept includes the storage device according to Claim 11, wherein the virtual I / O queue creation command includes a second shared namespace attribute that specifies a second array of namespaces that share access to a range of LBAs.

[0193] Claim 22. An embodiment of the inventive concept includes a storage device according to Claim 2, wherein the FPGA further includes doorbell distribution logic for locating a first virtual doorbell of a first virtual I / O queue in a memory page different from a second doorbell of a second virtual I / O queue.

[0194] Claim 23. An embodiment of the inventive concept includes a field programmable gate array (FPGA) including:

[0195] at least one virtual input / output (I / O) queue for requests from at least one virtual machine (VM) on a host device; and

[0196] mapping logic for mapping a virtual I / O queue of the at least one virtual I / O queue to an I / O queue on a storage device such that an I / O request received from a VM in the virtual I / O queue is passed to the storage device via the I / O queue, and a result received from the storage device in the I / O queue is passed to the VM via the virtual I / O queue,

[0197] wherein the FPGA supports a virtual I / O queue creation command to request allocation of a virtual I / O queue of the at least one virtual I / O queue for a VM of the at least one VM, the virtual I / O queue creation command including an LBA range attribute of a range of logical block addresses (LBAs) to be associated with the virtual I / O queue, and

[0198] wherein a storage device separate from but connected to the FPGA maps the range of LBAs to a range of physical block addresses (PBAs) in the storage device.

[0199] Claim 24. An embodiment of the inventive concept includes an FPGA according to Claim 23, wherein the virtual I / O queue includes a submission queue and a completion queue.

[0200] Claim 25. An embodiment of the inventive concept includes an FPGA according to Claim 23, wherein the FPGA receives a virtual I / O queue creation command from a hypervisor of a VM on a host device.

[0201] Claim 26. An embodiment of the inventive concept includes an FPGA according to Claim 23, wherein the LBA range attribute includes a start LBA and an end LBA.

[0202] Claim 27. An embodiment of the inventive concept includes an FPGA according to Claim 23, wherein:

[0203] the LBA range attribute includes a start LBA; and

[0204] the virtual I / O queue creation command further includes a queue size.

[0205] Claim 28. An embodiment of the inventive concept includes the FPGA according to Claim 23, wherein the virtual I / O queue creation command includes a quality-of-service (QoS) attribute for a QoS parameter for the VM.

[0206] Claim 29. An embodiment of the inventive concept includes the FPGA according to Claim 28, wherein the QoS attribute is extracted from a set including a minimum bandwidth, a maximum bandwidth, a minimum number of read requests per second, a maximum number of read requests per second, a minimum number of read bytes per second, a maximum number of read bytes per second, a minimum number of write requests per second, a maximum number of write requests per second, a minimum number of write bytes per second, and a maximum number of write bytes per second.

[0207] Claim 30. An embodiment of the inventive concept includes the FPGA according to Claim 23, wherein the virtual I / O queue creation command includes a shared namespace attribute that specifies an array of namespaces that share access to a range of LBAs.

[0208] Claim 31. An embodiment of the inventive concept includes the FPGA according to Claim 23, wherein the FPGA maps a plurality of virtual I / O queues to an I / O queue on a storage device.

[0209] Claim 32. An embodiment of the inventive concept includes the FPGA according to Claim 31, wherein the FPGA is operable to invoke an I / O queue creation command on the storage device to create the I / O queue.

[0210] Claim 33. An embodiment of the inventive concept includes the FPGA according to Claim 32, wherein the I / O queue creation command includes a second LBA attribute for a second range of LBAs to be associated with the I / O queue.

[0211] Claim 34. An embodiment of the inventive concept includes the FPGA according to Claim 23, further including doorbell distribution logic for locating a first virtual doorbell of the virtual I / O queue in a memory page different from a second doorbell of a second virtual I / O queue.

[0212] Claim 35. An embodiment of the inventive concept includes a method, including:

[0213] Receiving, at a storage device, an I / O queue creation command for a first VM on a host device, the I / O queue creation command including at least an LBA range attribute for a range of logical block addresses (LBAs) to be associated with an I / O queue on the storage device;

[0214] Establishing an I / O queue on the storage device;

[0215] Selecting, on the storage device, a range of physical block addresses (PBAs) that is at least as large as the range of LBAs;

[0216] Map a range of LBAs to a range of PBAs; and

[0217] Return a success indicator,

[0218] whereby a second VM on the host device is thereby denied access to the range of PBAs.

[0219] Claim 36. An embodiment of the inventive concept includes the method according to claim 35, wherein:

[0220] The storage device includes a solid state drive (SSD) storage device; and

[0221] The SSD storage device uses a non-volatile memory express (NVMe) interface to the host device.

[0222] Claim 37. An embodiment of the inventive concept includes the method according to claim 36, wherein the I / O queue includes a submission queue and a completion queue.

[0223] Claim 38. An embodiment of the inventive concept includes the method according to claim 36, wherein receiving an I / O queue creation command from a first VM on the host device includes receiving the I / O queue creation command from a hypervisor of the first VM on the host device.

[0224] Claim 39. An embodiment of the inventive concept includes the method according to claim 36, wherein receiving an I / O queue creation command from a first VM on the host device includes receiving the I / O queue creation command from a field programmable gate array (FPGA) of the first VM on the host device.

[0225] Claim 40. An embodiment of the inventive concept includes the method according to claim 39, wherein the FPGA is included in the storage device.

[0226] Claim 41. An embodiment of the inventive concept includes the method according to claim 39, wherein the FPGA is separate from but connected to the storage device.

[0227] Claim 42. An embodiment of the inventive concept includes the method according to claim 36, wherein the LBA range attribute includes a start LBA and an end LBA.

[0228] Claim 43. An embodiment of the inventive concept includes the method according to claim 36, wherein:

[0229] The LBA range attribute includes a start LBA; and

[0230] The I / O queue creation command further includes a queue size.

[0231] Claim 44. An embodiment of the inventive concept includes the method according to Claim 36, wherein the I / O queue creation command includes a quality of service (QoS) attribute for the quality of service parameters of a first VM.

[0232] Claim 45. An embodiment of the inventive concept includes the method according to Claim 44, wherein the QoS attribute is extracted from a set including a minimum bandwidth, a maximum bandwidth, a minimum number of read requests per second, a maximum number of read requests per second, a minimum number of read bytes per second, a maximum number of read bytes per second, a minimum number of write requests per second, a maximum number of write requests per second, a minimum number of write bytes per second, and a maximum number of write bytes per second.

[0233] Claim 46. An embodiment of the inventive concept includes the method according to Claim 36, wherein the I / O queue creation command includes a shared namespace attribute that specifies an array of namespaces that share access to a range of LBAs.

[0234] Claim 47. An embodiment of the inventive concept includes the method according to Claim 36, further comprising:

[0235] identifying a plurality of doorbells associated with a plurality of I / O queues;

[0236] assigning each of the plurality of doorbells to a memory address among a first plurality of memory addresses in a storage device, the first plurality of memory addresses being distributed across a plurality of memory pages;

[0237] requesting an address space from a host device;

[0238] mapping the first plurality of memory addresses to a second plurality of memory addresses in the address space, the second plurality of memory addresses being distributed across a plurality of memory pages; and

[0239] providing at least the doorbell addresses of the second plurality of memory addresses to a first VM.

[0240] Claim 48. An embodiment of the inventive concept includes the method according to Claim 47, further comprising:

[0241] receiving a doorbell request to access a doorbell address;

[0242] mapping the doorbell address back to an address among the first plurality of memory addresses; and

[0243] sending the doorbell request to the address among the first plurality of memory addresses.

[0244] Claim 49. An embodiment of the inventive concept includes a method, comprising:

[0245] Receiving, at a field programmable gate array (FPGA), a virtual I / O queue creation command from a hypervisor of a first virtual machine (VM) on a host device, the virtual I / O queue creation command including at least a logical block address (LBA) range attribute of a range of LBAs to be associated with an I / O queue on a storage device;

[0246] Establishing a first virtual I / O queue on the FPGA;

[0247] Sending an I / O queue creation command to the storage device to establish an I / O queue on the storage device;

[0248] Receiving a result from the storage device;

[0249] Mapping the first virtual I / O queue to the I / O queue;

[0250] Associating the range of LBAs with the first virtual I / O queue; and

[0251] Returning a success indicator to the hypervisor,

[0252] wherein the range of LBAs is mapped to a range of physical block addresses (PBAs) on the storage device, and

[0253] wherein a second VM on the host device is thereby denied access to the range of PBAs.

[0254] Claim 50. An embodiment of the inventive concept includes the method according to Claim 49, wherein the first virtual I / O queue includes a submission queue and a completion queue.

[0255] Claim 51. An embodiment of the inventive concept includes the method according to Claim 49, wherein the FPGA maps both the first virtual I / O queue and a second virtual I / O queue to the I / O queue.

[0256] Claim 52. An embodiment of the inventive concept includes the method according to Claim 49, wherein the I / O queue creation command includes an LBA range attribute of a range of LBAs.

[0257] Claim 53. An embodiment of the inventive concept includes the method according to Claim 49, wherein the LBA range attribute includes a start LBA and an end LBA.

[0258] Claim 54. An embodiment of the inventive concept includes the method according to Claim 49, wherein:

[0259] the LBA range attribute includes a start LBA; and

[0260] the virtual I / O queue creation command further includes a queue size.

[0261] Claim 55. An embodiment of the inventive concept includes the method according to Claim 49, wherein the virtual I / O queue creation command includes a quality of service (QoS) attribute for the quality of service parameters of a first VM.

[0262] Claim 56. An embodiment of the inventive concept includes the method according to Claim 55, wherein the QoS attribute is extracted from a set including a minimum bandwidth, a maximum bandwidth, a minimum number of read requests per second, a maximum number of read requests per second, a minimum number of read bytes per second, a maximum number of read bytes per second, a minimum number of write requests per second, a maximum number of write requests per second, a minimum number of write bytes per second, and a maximum number of write bytes per second.

[0263] Claim 57. An embodiment of the inventive concept includes the method according to Claim 49, wherein the virtual I / O queue creation command includes a shared namespace attribute that specifies an array of namespaces that share access to a range of LBAs.

[0264] Claim 58. An embodiment of the inventive concept includes the method according to Claim 49, further comprising:

[0265] identifying a plurality of doorbells associated with a plurality of I / O queues;

[0266] assigning each of the plurality of doorbells to a memory address among a first plurality of memory addresses in an FPGA, the first plurality of memory addresses being distributed across a plurality of memory pages;

[0267] requesting an address space from a host device;

[0268] mapping the first plurality of memory addresses to a second plurality of memory addresses in the address space, the second plurality of memory addresses being distributed across a plurality of memory pages; and

[0269] providing at least the doorbell addresses of the second plurality of memory addresses to a first VM.

[0270] Claim 59. An embodiment of the inventive concept includes the method according to Claim 58, further comprising:

[0271] receiving a doorbell request to access a doorbell address;

[0272] mapping the doorbell address back to an address among the first plurality of memory addresses; and

[0273] sending the doorbell request to the address among the first plurality of memory addresses.

[0274] Claim 60. An embodiment of the inventive concept includes a method, comprising:

[0275] receiving, from a virtual machine (VM) on a host device, a first request destined for a storage device;

[0276] Capture the first request to prevent it from reaching the storage device;

[0277] Send a second request to a field programmable gate array (FPGA), the second request simulating the first request;

[0278] Receive the result of the second request from the FPGA; and

[0279] Send the result of the second request to the VM.

[0280] Claim 61. An embodiment of the inventive concept includes the method according to Claim 60, wherein:

[0281] The first request includes a first PCI configuration space request to access a first PCI configuration space; and

[0282] The second request includes a second PCI configuration space request to access a second PCI configuration space.

[0283] Claim 62. An embodiment of the inventive concept includes the method according to Claim 60, wherein:

[0284] The first request includes an I / O queue creation request to create an I / O queue for a storage device; and

[0285] The second request includes a virtual I / O queue creation request to create a virtual I / O queue for the FPGA.

[0286] Claim 63. An embodiment of the inventive concept includes the method according to Claim 62, wherein the second request further includes a logical block address (LBA) attribute of a range of LBAs to be associated with the virtual I / O queue.

[0287] Claim 64. An embodiment of the inventive concept includes the method according to Claim 63, wherein the virtual I / O queue creation request includes a shared namespace attribute that specifies an array of namespaces that share access to the range of LBAs.

[0288] Claim 65. An embodiment of the inventive concept includes the method according to Claim 62, wherein the second request further includes a quality of service (QoS) attribute for QoS parameters for the VM.

[0289] Claim 66. An embodiment of the inventive concept includes the method according to Claim 65, wherein the QoS attribute is extracted from a set including minimum bandwidth, maximum bandwidth, minimum number of read requests per second, maximum number of read requests per second, minimum number of read bytes per second, maximum number of read bytes per second, minimum number of write requests per second, maximum number of write requests per second, minimum number of write bytes per second, and maximum number of write bytes per second.

[0290] Claim 67. An embodiment of the inventive concept includes an article including a non-transitory storage medium storing instructions that, when executed by a machine, cause:

[0291] Receiving, at a storage device, an I / O queue creation command for a first VM on a host device, the I / O queue creation command including at least an LBA range attribute of a range of logical block addresses (LBAs) to be associated with an I / O queue on the storage device;

[0292] Establishing an I / O queue on the storage device;

[0293] Selecting, on the storage device, a range of physical block addresses (PBAs) that is at least as large as the range of LBAs;

[0294] Mapping the range of LBAs to the range of PBAs; and

[0295] Returning a success indicator,

[0296] wherein a second VM on the host device is thereby denied access to the range of PBAs.

[0297] Claim 68. An embodiment of the inventive concept includes the article according to Claim 67, wherein:

[0298] The storage device includes a solid state drive (SSD) storage device; and

[0299] The SSD storage device uses a non-volatile memory express (NVMe) interface to the host device.

[0300] Claim 69. An embodiment of the inventive concept includes the article according to Claim 68, wherein the I / O queue includes a submission queue and a completion queue.

[0301] Claim 70. An embodiment of the inventive concept includes the article according to Claim 68, wherein receiving the I / O queue creation command for the first VM on the host device includes receiving the I / O queue creation command from a hypervisor of the first VM on the host device.

[0302] Claim 71. An embodiment of the inventive concept includes the article according to Claim 68, wherein receiving the I / O queue creation command for the first VM on the host device includes receiving the I / O queue creation command from a field programmable gate array (FPGA) of the first VM on the host device.

[0303] Claim 72. An embodiment of the inventive concept includes the article according to Claim 71, wherein the FPGA is included in the storage device.

[0304] Claim 73. An embodiment of the inventive concept includes the article according to Claim 71, wherein the FPGA is separated from but connected to the storage device.

[0305] Claim 74. An embodiment of the inventive concept includes the article according to Claim 68, wherein the LBA range attribute includes a start LBA and an end LBA.

[0306] Claim 75. An embodiment of the inventive concept includes the article according to Claim 68, wherein:

[0307] the LBA range attribute includes a start LBA; and

[0308] the I / O queue creation command further includes a queue size.

[0309] Claim 76. An embodiment of the inventive concept includes the article according to Claim 68, wherein the I / O queue creation command includes a quality of service (QoS) attribute for QoS parameters of a first VM.

[0310] Claim 77. An embodiment of the inventive concept includes the article according to Claim 76, wherein the QoS attribute is extracted from a set including a minimum bandwidth, a maximum bandwidth, a minimum number of read requests per second, a maximum number of read requests per second, a minimum number of read bytes per second, a maximum number of read bytes per second, a minimum number of write requests per second, a maximum number of write requests per second, a minimum number of write bytes per second, and a maximum number of write bytes per second.

[0311] Claim 78. An embodiment of the inventive concept includes the article according to Claim 68, wherein the I / O queue creation command includes a shared namespace attribute that specifies an array of namespaces that share access to a range of LBAs.

[0312] Claim 79. An embodiment of the inventive concept includes the article according to Claim 68, further including:

[0313] identifying a plurality of doorbells associated with a plurality of I / O queues;

[0314] assigning each of the plurality of doorbells to a memory address among a first plurality of memory addresses in the storage device, the first plurality of memory addresses being distributed across a plurality of memory pages;

[0315] requesting an address space from a host device;

[0316] mapping the first plurality of memory addresses to a second plurality of memory addresses in the address space, the second plurality of memory addresses being distributed across a plurality of memory pages; and

[0317] providing at least a doorbell address of the second plurality of memory addresses to the first VM.

[0318] Claim 80. An embodiment of the inventive concept includes the article according to Claim 79, further comprising:

[0319] Receiving a doorbell request to access a doorbell address;

[0320] Mapping the doorbell address back to an address in the first plurality of memory addresses; and

[0321] Sending the doorbell request to the address in the first plurality of memory addresses.

[0322] Claim 81. An embodiment of the inventive concept includes an article including a non-transitory storage medium storing instructions that, when executed by a machine, cause:

[0323] Receiving, at a field programmable gate array (FPGA), from a hypervisor of a first VM on a host device, a virtual I / O queue creation command that includes at least an LBA range attribute of a range of logical block addresses (LBAs) to be associated with an I / O queue on a storage device;

[0324] Establishing a first virtual I / O queue on the FPGA;

[0325] Sending an I / O queue creation command to the storage device to establish an I / O queue on the storage device;

[0326] Receiving a result from the storage device;

[0327] Mapping the first virtual I / O queue to the I / O queue;

[0328] Associating the range of LBAs with the first virtual I / O queue; and

[0329] Returning a success indicator to the hypervisor,

[0330] wherein the range of LBAs is mapped to a range of physical block addresses (PBAs) on the storage device, and

[0331] wherein a second VM on the host device is thereby denied access to the range of PBAs.

[0332] Claim 82. An embodiment of the inventive concept includes the article according to Claim 81, wherein the first virtual I / O queue includes a submission queue and a completion queue.

[0333] Claim 83. An embodiment of the inventive concept includes the article according to Claim 81, wherein the FPGA maps both the first virtual I / O queue and a second virtual I / O queue to the I / O queue.

[0334] Claim 84. An embodiment of the inventive concept includes an article according to Claim 81, wherein the I / O queue creation command includes an LBA range attribute of a range of LBAs.

[0335] Claim 85. An embodiment of the inventive concept includes an article according to Claim 81, wherein the LBA range attribute includes a start LBA and an end LBA.

[0336] Claim 86. An embodiment of the inventive concept includes an article according to Claim 81, wherein:

[0337] the LBA range attribute includes a start LBA; and

[0338] the virtual I / O queue creation command further includes a queue size.

[0339] Claim 87. An embodiment of the inventive concept includes an article according to Claim 81, wherein the virtual I / O queue creation command includes a quality of service (QoS) attribute for QoS parameters of a first VM.

[0340] Claim 88. An embodiment of the inventive concept includes an article according to Claim 87, wherein the QoS attribute is extracted from a set including a minimum bandwidth, a maximum bandwidth, a minimum number of read requests per second, a maximum number of read requests per second, a minimum number of read bytes per second, a maximum number of read bytes per second, a minimum number of write requests per second, a maximum number of write requests per second, a minimum number of write bytes per second, and a maximum number of write bytes per second.

[0341] Claim 89. An embodiment of the inventive concept includes an article according to Claim 81, wherein the virtual I / O queue creation command includes a shared namespace attribute that specifies an array of namespaces that share access to a range of LBAs.

[0342] Claim 90. An embodiment of the inventive concept includes an article according to Claim 81, further including:

[0343] identifying a plurality of doorbells associated with a plurality of I / O queues;

[0344] assigning each of the plurality of doorbells to a memory address among a first plurality of memory addresses in an FPGA, the first plurality of memory addresses being distributed across a plurality of memory pages;

[0345] requesting an address space from a host device;

[0346] mapping the first plurality of memory addresses to a second plurality of memory addresses in the address space, the second plurality of memory addresses being distributed across a plurality of memory pages; and

[0347] providing at least doorbell addresses of the second plurality of memory addresses to a first VM.

[0348] Claim 91. An embodiment of the inventive concept includes the article according to Claim 90, further comprising:

[0349] Receiving a doorbell request to access a doorbell address;

[0350] Mapping the doorbell address back to an address in the first plurality of memory addresses; and

[0351] Sending the doorbell request to the address in the first plurality of memory addresses.

[0352] Claim 92. An embodiment of the inventive concept includes an article that includes a non-transitory storage medium having instructions stored thereon that, when executed by a machine, cause:

[0353] Receiving a first request from a virtual machine (VM) on a host device, the first request destined for a storage device;

[0354] Capturing the first request to prevent it from reaching the storage device;

[0355] Sending a second request to a field-programmable gate array (FPGA), the second request simulating the first request;

[0356] Receiving a result of the second request from the FPGA; and

[0357] Sending the result of the second request to the VM.

[0358] Claim 93. An embodiment of the inventive concept includes the article according to Claim 92, wherein:

[0359] The first request includes a first PCI configuration space request to access a first PCI configuration space; and

[0360] The second request includes a second PCI configuration space request to access a second PCI configuration space.

[0361] Claim 94. An embodiment of the inventive concept includes the article according to Claim 92, wherein:

[0362] The first request includes an I / O queue creation request to create an I / O queue for the storage device; and

[0363] The second request includes a virtual I / O queue creation request to create a virtual I / O queue for the FPGA.

[0364] Claim 95. An embodiment of the inventive concept includes the article according to Claim 94, wherein the second request further includes a logical block address (LBA) attribute of a range of LBAs to be associated with the virtual I / O queue.

[0365] Claim 96. An embodiment of the inventive concept includes an article according to Claim 95, wherein the virtual I / O queue creation request includes a shared namespace attribute that specifies an array of namespaces that share access to a range of LBAs.

[0366] Claim 97. An embodiment of the inventive concept includes an article according to Claim 94, wherein the second request further includes a quality of service (QoS) attribute for a quality of service parameter for the VM.

[0367] Claim 98. An embodiment of the inventive concept includes an article according to Claim 97, wherein the QoS attribute is extracted from a set including a minimum bandwidth, a maximum bandwidth, a minimum number of read requests per second, a maximum number of read requests per second, a minimum number of bytes read per second, a maximum number of bytes read per second, a minimum number of write requests per second, a maximum number of write requests per second, a minimum number of bytes written per second, and a maximum number of bytes written per second.

[0368] Accordingly, considering the various permutations of the embodiments described herein, this detailed description and the accompanying material are intended to be illustrative only and should not be regarded as limiting the scope of the inventive concept. Thus, what is claimed as the inventive concept are all such modifications as may fall within the scope and spirit of the appended claims and their equivalents.

Claims

1. A storage device for dynamically allocating physical storage device resources in a virtualized environment, comprising: a storage device for data; and at least one input / output I / O queue for requests from at least one virtual machine VM on a host device, wherein the storage device supports an I / O queue creation command to request an I / O queue for allocating the at least one I / O queue to the VM of the at least one VM, the I / O queue creation command including an LBA range attribute of a range of logical block addresses LBA to be associated with the I / O queue and a quality of service QoS attribute for a quality of service parameter of the VM, wherein the QoS attribute includes at least one of a minimum bandwidth, a maximum bandwidth, a minimum number of read requests per second, a maximum number of read requests per second, a minimum number of read bytes per second, a maximum number of read bytes per second, a minimum number of write requests per second, a maximum number of write requests per second, a minimum number of write bytes per second, and a maximum number of write bytes per second, wherein the storage device maps the range of LBA to a range of physical block addresses PBA in the storage device for data, wherein the storage device includes a solid state drive SSD storage device, and wherein the SSD storage device uses a non-volatile memory express NVMe interface to the host device.

2. The storage device according to claim 1, wherein, the storage device receives the I / O queue creation command from a hypervisor on the host device.

3. The storage device according to claim 1, wherein, the LBA range attribute includes a start LBA and an end LBA.

4. The storage device according to claim 1, wherein, the I / O queue creation command includes a shared namespace attribute that specifies an array of namespaces that share access to the range of LBA.

5. The storage device according to claim 1, further comprising a field programmable gate array FPGA that maps a plurality of virtual I / O queues to the I / O queue, wherein: the FPGA supports a virtual I / O queue creation command to request a first virtual I / O queue for allocating the plurality of virtual I / O queues, the virtual I / O queue creation command including a second LBA range attribute of a second range of logical block addresses LBA to be associated with the first virtual I / O queue, the first virtual I / O queue being associated with the I / O queue.

6. The storage device according to claim 1, further comprising doorbell distribution logic for locating a first doorbell of the I / O queue in a memory page different from a second doorbell of a second I / O queue.

7. A field programmable gate array FPGA for dynamically allocating physical storage device resources in a virtualized environment, comprising: at least one virtual input / output I / O queue for requests from at least one virtual machine VM on a host device; and Mapping logic for mapping virtual I / O queues of the at least one virtual I / O queue to I / O queues on a storage device such that I / O requests received from a VM in the virtual I / O queue are passed to the storage device via the I / O queue, and results received from the storage device in the I / O queue are passed to the VM via the virtual I / O queue, wherein the FPGA supports a virtual I / O queue creation command to request allocation of virtual I / O queues of the at least one virtual I / O queue for a VM of the at least one VM, the virtual I / O queue creation command including an LBA range attribute of a range of logical block addresses LBA to be associated with the virtual I / O queue and a quality of service attribute for quality of service QoS parameters for the VM, wherein the QoS attribute includes at least one of a minimum bandwidth, a maximum bandwidth, a minimum number of read requests per second, a maximum number of read requests per second, a minimum number of bytes read per second, a maximum number of bytes read per second, a minimum number of write requests per second, a maximum number of write requests per second, a minimum number of bytes written per second, and a maximum number of bytes written per second, and wherein a storage device separate from but connected to the FPGA maps the range of the LBA to a range of physical block addresses PBA in the storage device.

8. The FPGA according to claim 7, wherein the LBA range attribute includes a start LBA and an end LBA.

9. The FPGA according to claim 7, wherein the virtual I / O queue creation command includes a shared namespace attribute that specifies an array of namespaces that share access to the range of the LBA.

10. The FPGA according to claim 7, wherein the FPGA maps a plurality of virtual I / O queues to I / O queues on the storage device.

11. The FPGA according to claim 10, wherein the FPGA is operable to call an I / O queue creation command on the storage device to create the I / O queue.

12. The FPGA according to claim 7, further comprising doorbell distribution logic for locating a first virtual doorbell of the virtual I / O queue in a memory page different from a second doorbell of a second virtual I / O queue.

13. An article including a non-transitory storage medium for dynamically allocating physical storage device resources in a virtualized environment, the non-transitory storage medium storing instructions that, when executed by a machine, cause: Receiving a first request from a virtual machine VM on a host device, the first request going to a storage device; Capturing the first request to prevent it from reaching the storage device; Sending a second request to a field programmable gate array FPGA, the second request simulating the first request; Receiving a result of the second request from the FPGA; And Sending the result of the second request to the VM, wherein the first request includes an I / O queue creation request for creating an I / O queue for the storage device, wherein the second request includes a virtual I / O queue creation request for creating a virtual I / O queue for the FPGA and a quality of service (QoS) attribute for QoS parameters of the VM, and wherein the QoS attribute includes at least one of a minimum bandwidth, a maximum bandwidth, a minimum number of read requests per second, a maximum number of read requests per second, a minimum number of bytes read per second, a maximum number of bytes read per second, a minimum number of write requests per second, a maximum number of write requests per second, a minimum number of bytes written per second, and a maximum number of bytes written per second, wherein the second request further includes a logical block address (LBA) range attribute of a range of LBAs to be associated with the virtual I / O queue.

14. The article according to claim 13, wherein the virtual I / O queue creation request includes a shared namespace attribute that specifies an array of namespaces that share access to the range of LBAs.

Images