Method and apparatus for processing commands from a virtual machine

Abstract

A method and apparatus for processing commands from a virtual machine, the method comprising: accessing, by a virtual non-volatile memory device in a virtual machine monitor executing on one or more processors, a first command submitted to a guest queue by a native non-volatile memory driver executing in a guest virtual machine; generating, by the virtual non-volatile memory device, a translated command based on the first command by translating virtual parameters of the first command to physical parameters associated with a physical non-volatile memory device; submitting, by the virtual non-volatile memory device, the translated command to a shadow queue to be processed by the physical non-volatile memory device based on the physical parameters; and submitting, by the virtual non-volatile memory device, a completion status entry to the guest queue indicating completion of a direct memory access operation to copy data between the physical non-volatile memory device and a guest memory buffer corresponding to the guest virtual machine.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to memory in a processor system, and more particularly, to methods and apparatuses for processing commands from virtual machines. BACKGROUND

[0002] In a virtualized processing environment, a single physical platform is shared across multiple virtual machines (VMs) and / or virtual operating systems (OSs). Such virtualization employs a large number of physical resources as virtual resources allocated to different VMs. For example, these resources include central processing units (CPUs), storage (e.g., non-volatile data storage devices), memory (e.g., volatile random access memory (RAM)), graphics processing units (GPUs), network interface cards (NICs), etc. For storage devices, existing storage input-output (I / O) virtualization solutions are designed based on old hardware technologies, such as magnetic-based hard disk drive (HDD) storage and / or old slow NAND solid state drive (NAND-SSD) storage. BRIEF DESCRIPTION OF DRAWINGS

[0003] FIG. 1 illustrates an example existing paravirtualization (PV) block I / O service for providing VMs with access to physical non-volatile (NV) storage implemented as non-volatile memory express (NVMe) devices.

[0004] Figure 2 A host machine implementing example zero-copy block virtualization - mediated pass-through (ZCBV-MPT) techniques to provide VMs with access to physical NV storage is illustrated.

[0005] Figure 3 An example mediator and example virtual NVMe device that can be used to implement the ZCBV-MPT techniques described in connection with Figure 2 An example view of the non-volatile memory express (NVMe) protocol over Peripheral Component Interconnect Express (PCIe) bus for the example ZCBV-MPT techniques described.

[0006] Figure 4 An example mediator and example virtual NVMe device that facilitate performing zero-copy operations based on the example ZCBV-MPT techniques described in connection with Figure 2 Figure 2

[0007] Figure 5 An example guest queue 226a of a guest VM emulating PCI configuration and managing Figure 2 An example virtual NVMe device of the example ZCBV-MPT techniques described. Figure 2

[0008] Figure 6 An example mediator and example virtual NVMe device that facilitate performing zero-copy operations based on the example ZCBV-MPT techniques described in connection with Figure 2 ​​​Example client queue I / O commands for management Figure 2 Example shadow queues for implementing combination Figure 2 The example ZCBV-MPT technology Figure 2 Example virtual NVMe device.

[0009] Figure 7 This demonstrates management based on completed I / O commands. Figure 2 Example shadow queues and example client queues are provided to achieve combination. Figure 2 The example ZCBV-MPT technology described Figure 2 Example virtual NVMe device.

[0010] Figure 8 The executable is shown to define Figure 2 and Figures 4-6 The interface of the virtual NVMe device is used to achieve integration. Figure 2 The example machine-readable instructions for the ZCBV-MPT technology are described.

[0011] Figure 9 The executable is shown to define Figure 2 and Figures 4-6 The functionality of virtual NVMe devices to achieve integration Figure 2 The example machine-readable instructions for the ZCBV-MPT technology are described.

[0012] Figure 10 The illustration shows a host machine implementing an example of Zero Copy Block Virtualization – Paravirtualized I / O (ZCBV-PVIO) technology to provide VMs with access to physical NV memory.

[0013] Figure 11 This indicates that it can be executed to achieve the combination. Figures 2-9 The flowchart illustrates an example machine-readable instruction for the example ZCBV-MPT technology.

[0014] Figure 12 This indicates that it can be executed to achieve the combination. Figure 10 The flowchart illustrates an example machine-readable instruction for the example ZCBV-PVIO technology.

[0015] Figure 13 It is capable of executing by Figure 6 , Figure 7 , Figure 8 , Figure 9 and / or Figure 11 The example machine-readable instructions represent the implementation of the example ZCBV-MPT technology and / or the ability to execute the instructions disclosed herein. Figure 12 The example machine-readable instructions shown are for an example processor platform that implements the example ZCBV-PVIO technology disclosed herein.

[0016] The drawings are not to scale. Whenever possible, like reference numerals will be used throughout the drawing(s) and accompanying written description to refer to like or similar parts. DETAILED DESCRIPTION

[0017] Examples disclosed herein can be used to process commands from virtual machines using techniques that improve virtualization performance associated with accessing virtualized storage and memory space. Examples disclosed herein are described in connection with virtualization of non-volatile memory express (NVMe) devices. NVMe devices are data storage devices that communicate with a host via the NVMe protocol and use non-volatile memory (e.g., memory devices using chalcogenide glass, single- or multi-threshold level NAND flash, NOR flash, 3D flash, three-dimensional (3D) cross-point memory, ferroelectric transistor random access memory (FeTRAM or FeRAM), multi-level phase change memory (PRAM, PCM), anti-ferroelectric memory, magnetoresistive random access memory (MRAM) including memristor technology, resistive memory including metal-oxide based, oxygen-vacancy based, and conductive-bridge random access memory (CB-RAM), or spin-transfer torque (STT)-MRAM, spintronic magnetic junction memory-based devices, magnetic tunnel junction (MTJ)-based devices, DW (domain wall) and SOT (spin orbit transfer)-based devices, thyristor-based memory devices, non-volatile RAM (NVRAM), resistive random access memory (ReRAM), resistive memory, nanowire memory, or a combination of any of the above or other memory). The NVMe protocol is a high-performance, scalable host controller interface developed by NVM Express, Inc. for use with PCI Express (PCIe) storage devices that use NAND flash, NOR flash, 3D flash, 3D cross-point memory, FeTRAM, PRAM, anti-ferroelectric memory, MRAM, CB-RAM, FeRAM, STT-MRAM, spintronic magnetic junction memory-based devices, MTJ-based devices, DW and SOT-based devices, thyristor-based memory devices, NVRAM, ReRAM, resistive memory, nanowire memory, or a combination of any of the above or other memory. Enterprise and / or client systems using solid state storage. The NVMe interface is typically used for fast storage I / O. With NVMe, an operating system (OS) can issue I / O requests by placing DMA requests in an I / O queue, and the NVMe driver can utilize multiple I / O queues (e.g., one for each CPU core) to issue I / O requests to the storage device. The NVMe driver can also utilize multiple I / O queues to issue I / O requests to multiple storage devices. Optane TMThe device supports 16 I / O queues) to service multiple I / O requests using parallel I / O processing. However, the examples disclosed herein can be implemented in conjunction with any other type of NV memory device using any other type of host controller interface, bus interface, and / or transport protocol. For example, the example techniques disclosed herein can be adapted for use with the Serial Advanced Technology Attachment (SATA) Express (SATAe) bus interface protocol and / or the mini-SATA (mSATA) bus interface protocol defined by the Serial ATA International Organization. Additionally or alternatively, the example techniques disclosed herein can be adapted for use with the Serial Attached, Small Computer System Interface (SCSI) protocol, also known as the SAS bus interface protocol, defined by the International Committee for Information Technology Standards (INCITS). In yet another example, the example techniques disclosed herein can be adapted for use with the Advanced Host Controller Interface (AHCI) bus interface protocol defined by Intel Corporation. Additionally or alternatively, the techniques disclosed herein can be adapted for use with any other suitable bus interface standard currently available and / or developed in the future.

[0018] Virtualization technology involves a single physical platform hosting multiple guest virtual machines (VMs). To allocate usage of hardware resources such as central processing units (CPUs), network interface cards (NICs), storage, memory, graphics processing units (GPUs), etc., several virtualization technologies have been developed to enable virtualization of such physical hardware resources into allocable virtual resources. For example, a single physical CPU can be allocated as multiple virtual CPUs to different VMs. Each VM identifies the corresponding virtual CPU(s) as its own CPU(s), but in fact each VM uses only a portion of the same underlying physical CPU that is also used by other VMs. Similarly, a single physical storage device can be allocated as multiple virtual data stores to different VMs. Each VM can independently access its allocated virtual data store space without dependency on other VMs, but all VMs access the same underlying physical storage device by using portions of the physical storage device that are isolated or partitioned from other portions of the physical storage device.

[0019] Existing data storage virtualization technology is based on older hardware technology such as magnetic-based hard disk drive (HDD) storage and / or older slow NAND solid state drive (NAND-SSD) storage. As such, existing data storage virtualization technology is based on the capabilities of existing storage devices that operate slower than more recently developed storage devices. As such, when newer storage devices are used with older virtual systems, data access performance is limited by existing data storage virtualization technology developed based on existing, slower storage devices.

[0020] FIG. 1 illustrates an existing PV block I / O service for providing a guest VM 102 with access to physical NV memory 104 implemented as an NVMe device. The existing PV block I / O service of FIG. 1 is a storage / block I / O virtualization solution. In FIG. 1, the guest VM 102 runs a paravirtualized driver represented as a front-end (FE) block driver 106. A paravirtualized driver is a driver that is able to directly access hardware via a native device driver without an intermediary host operating system (OS) to emulate the hardware for the paravirtualized driver. Unlike paravirtualization, full virtualization uses a fully virtualized driver that executes in the guest VM. Such a fully virtualized driver invokes virtual hardware that is emulated by the host OS. The host OS emulates the hardware, forwards the invocation to the underlying physical device via a native device driver, receives a response from the physical device, and forwards the response to the virtualized driver of the guest VM via the emulated hardware.

[0021] In FIG. 1, the guest VM 102 can be implemented using a kernel-based virtual machine (KVM), and the FE block driver 106 can be implemented using a virtio block driver that communicates with a back-end (BE) block service 108 running in an input / output virtual machine (IOVM) 110 executing as a virtual machine monitor (VMM) 112 (e.g., hypervisor) on the host OS (or service OS) shown in FIG. 1. Virtio is a standard for virtualization of network and disk device drivers in a paravirtualized hypervisor (e.g., VMM 112) in which only the device driver of the guest VM (e.g., FE block driver 106 of FIG. 1) “knows” that it is running in a virtual environment. The FE block driver 106 relies on the host-side file system 114 and / or native block system driver 116 of the IOVM 110 to read from and / or write to / read and / or write to storage implemented by the physical NV memory 104.

[0022] The example process flow of the existing PV block I / O service of FIG. 1 involves the FE block driver 106 placing a read request in the shared ring buffer 120 for access and processing by the BE block service 108. Based on the read request accessed by the BE block service 108, the IOVM 110 allocates a host memory buffer, and the native block system driver 116 sends a request to the physical NV memory 104 to read the requested data into the host memory buffer. The physical NV memory 104 performs a direct memory access (DMA) to write the requested data into the host memory buffer. Then, as shown by reference numeral 124, the BE block service 108 copies the data from the host memory buffer in the IOVM 110 to a guest memory buffer via the shared ring buffer 120. The FE block driver 106 can then access the requested data from the guest memory buffer. However, the data copy 124 is performed by a CPU of the host machine 126 on which the VMM 112 runs. As a result, during the memory intensive copy operation, the CPU resources of the host machine 126 can become overloaded, causing a decrease in performance of other processes of the host machine 126.

[0023] The resource and latency cost of such data copying 124 of FIG. 1 for block devices such as NV memory is generally acceptable for traditional hardware in which the bandwidth of hard disk drives (HDDs) and / or slow NAND solid state drives (SSDs) is at most 100s of MB / s. For such traditional hardware, the guest VMs are still able to achieve near maximum throughput of the underlying physical storage devices by trading off CPU cycles needed to perform the data request and copying. However, when using newer, faster storage devices in the host such as fast NAND-SSD and / or phase change memory (PCM) based devices that can achieve data transfer speeds of 2 GB / s, such processing and time resources become a big problem. Using the prior art of FIG. 1 with such newer, faster storage devices results in saturation of the processor resources, limiting the virtualization performance (guest throughput vs. physical throughput) of the virtual system. Additionally, the latency impact of the virtualization overhead associated with the prior art of FIG. 1 when used with such newer, faster storage devices will negatively impact performance. Optane TM devices) can result in saturation of the processor resources, limiting the virtualization performance (guest throughput vs. physical throughput) of the virtual system. Additionally, the latency impact of the virtualization overhead associated with the prior art of FIG. 1 when used with such newer, faster storage devices will negatively impact performance.

[0024] Examples disclosed herein improve virtualization performance associated with accessing storage and / or memory space. Examples disclosed herein include zero-copy block virtualization - mediated pass-through (ZCBV-MPT) techniques and zero-copy block virtualization - paravirtualized I / O (ZCBV-PV IO) techniques. In example ZCBV-MPT techniques, a guest VM runs a native NVMe device driver to access a physical NVMe storage by placing data access requests in a guest queue. Example ZCBV-MPT techniques also involve a VMM managing a shadow queue corresponding to the guest queue. To improve data access performance, the shadow queue can be executed directly in a hardware NVMe storage controller so that the NVMe device can perform DMA operations to directly copy requested data between the NVMe storage space and a guest memory space that is now accessible by the native NVMe device driver running in the guest VM without being intercepted by the VMM in a bulk data path. Further details of ZCBV-MPT examples are described below in connection with Figures 2-9 and Figure 11 Further details of ZCBV-MPT examples are described below in connection with

[0025] Example ZCBV-PV IO involves using a PV block IO (PV-IO) driver to directly perform DMA operations between an NVMe storage device and a guest block buffer accessible by a guest VM. In example ZCBV-PV IO techniques, a guest VM executes a PV-IO driver. In some examples, the PV-IO driver can be implemented using a KVM virtio driver. In examples disclosed herein, the PV-IO driver leverages an optimized I / O interface for virtualization that extends an NVMe driver on the IO VM (or host) side. The PV-IO driver directly manages a shadow queue using guest memory buffers and executes the shadow queue in a physical NVMe device to perform DMA operations to directly copy data between the NVMe device and the guest memory buffers without performing data copy operations using the IO VM. Further details of ZCBV-PV IO examples are described below in connection with Figure 10 and Figure 12 Further details of ZCBV-PV IO examples are described below in connection with

[0026] Example ZCBV techniques disclosed herein eliminate the need to perform data copy operations on the VMM backend side of a virtualization system. As a result, example ZCBV techniques disclosed herein improve the efficiency of block device I / O virtualization. In addition to reducing the use of CPU cycles (e.g., for performing copy operations of bulk data between NVMe storage and guest VM memory space), example ZCBV techniques disclosed herein also improve responsiveness (e.g., reduce latency) of virtual resources (e.g., virtual data storage resources based on underlying physical NVMe data storage resources).

[0027] Figure 2 The illustration depicts an example guest VM (shown as guest VM-A 202a and guest VM-B 202b) and an example host machine 204 implementing the example ZCBV-MPT technology to provide VMs 202a and 202b with access to example physical NV memory 206. The examples disclosed herein can be implemented using any number of guest VMs. The physical NV memory 206, implemented using NVMe device 206, is described in conjunction with this. Figure 2 Examples are provided. However, the example ZCBV-MPT technology can be implemented alternatively by combining other types of NV memory. The example ZCBV-MPT technology improves data access performance without sacrificing the ability of multiple guest VMs to share a single physical NVMe device. The example ZCBV-MPT technology involves executing native NVMe device drivers in the guest VMs and initiating direct memory access (DMA) copy operations for performance-critical I / O commands (e.g., data access requests). The example ZCBV-MPT technology also intercepts management commands that may affect global behavior across multiple guest VMs, management commands that do not need to be processed by the underlying physical hardware, and / or management commands that do not require the same high-performance processing as performance-critical I / O commands. The example ZCBV-MPT technology improves the performance of block I / O virtualization by using guest queues and corresponding shadow queues as described below. Lab tests of the example implementation of the ZCBV-MPT technology show a performance improvement of 100%+% compared to existing I / O paravirtualization technologies.

[0028] exist Figure 2 In the illustrated example, the example VMM 208 runs on host machine 204. Example VMM 208 can be implemented using any suitable host OS and / or hypervisor. For example, VMM 208 can be implemented using a host Linux / KVM OS. Example host machine 204 can be any physical computer or server. In the illustrated example, host machine 204 includes NVMe device 206 and example volatile memory 210, or is circuitically connected to NVMe device 206 and example volatile memory 210. Example volatile memory 210 can be implemented using any suitable random access memory (such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR) SDRAM, static RAM (SRAM), etc.).

[0029] In the illustrated example, each guest VM 202a, 202b executes a corresponding guest-native NVMe driver 214a, 214b. Also, in the illustrated example, VMM 208 executes an example guest queue manager 216, an example mediator 218, an example shadow queue manager 220, and an example host-native NVMe driver 222. In the illustrated example, NVMe drivers 214a, 214b, 222 are identified as native because I / O function calls programmed therein are structured to directly interface with a physical hardware device, such as NVMe device 206 (e.g., directly interface with firmware of NVMe device 206). In the illustrated example, guest-native NVMe drivers 214a, 214b are identical to host-native NVMe driver 222. As such, each of native NVMe drivers 214a, 214b, 222 operates as if it is directly interfacing with a physical NVMe device, even though only host-native NVMe driver 222 is directly interfacing with physical NVMe device 206.

[0030] In the illustrated example, guest queue manager 216, mediator 218, and shadow queue manager 220 implement a virtual NVMe device 224. NVMe device 224 is identified as virtual because it appears to interface with guest-native NVMe drivers 214a, 214b as if it is a physical hardware. As such, when guest-native NVMe drivers 214a, 214b communicate with NVMe device 224, guest-native NVMe drivers 214a, 214b behave as if they are communicating with physical hardware. However, NVMe device 224 operates in the context of "knowing" that it is not physical hardware and that it does not directly access physical hardware (e.g., NVMe device 206). In some examples, virtual NVMe device 224 can be implemented using a QEMU-hosted hypervisor to perform hardware virtualization. Example virtual NVMe device 224 translates data access requests from guest VMs 202a, 202b into data access requests suitable for service by NVMe device 206 to provide guest VMs 202a, 202b with data requested from NVMe device 206.

[0031] In other examples, alternatively, the client native NVMe drives 214a, 214b and the host native NVMe drive 222 can be any other suitable native non-volatile memory drive, any other suitable native memory drive, and / or any other suitable native hardware drive corresponding to the physical resource in which data is being accessed. Similarly, in other examples, alternatively, the virtual NVMe device 224 can be any other suitable virtual non-volatile memory, any other suitable virtual memory, and / or any other suitable virtual hardware corresponding to the physical resource in which data is being accessed. For example, although... Figure 2 The physical resource corresponding to the illustrated example is NVMe device 206, but in other examples, the corresponding physical resource can be any other type of non-volatile memory, any other type of memory, and / or any other type of physical hardware resource suitable for use with the examples disclosed herein. In addition to or alternative to the NVMe interface standard, the examples disclosed herein can also be implemented in conjunction with other suitable interface standards. For example, the techniques disclosed herein can be used with different types of bus interface standards (e.g., SATAe bus interface protocol, mSATA bus interface protocol, SAS bus interface protocol, AHCI bus interface protocol, etc.) to increase the data transfer speed associated with data access requests from guest VMs. For example, laboratory tests of the examples disclosed herein show that combining the examples disclosed herein with (e.g., in...) Optane TM Using a 3D cross-point memory implemented in memory can achieve data transfer speeds equal to or greater than 2000 megabytes per second (MB / s). This is significantly faster than transferring data at lower speeds (e.g., 1350 MB / s) from... Optane TM The examples disclosed herein improve upon existing techniques for reading data from memory. In other implementations, the example techniques disclosed herein can be used to improve the data transfer speed of guest VMs accessing data in other types of memory and / or via other types of interface standards.

[0032] An advantage of emulating one or more physical resources using the virtual NVMe device 224 is that the guest native NVMe drivers 214a, 214b in the guest VMs 202a, 202b do not need to be modified to be different or operate differently from the host native NVMe driver 222. That is, the guest native NVMe drivers 214a, 214b can be the same as the host native NVMe driver 222 because the guest native NVMe drivers 214a, 214b operate as if they are directly interfacing with the physical NVMe device (e.g., the NVMe device 206) that is being emulated by the virtual NVMe device 224. As such, by using native NVMe drivers (e.g., the guest native NVMe drivers 214a, 214b and / or a copy of the host native NVMe driver 222) in such additional guest VMs, the examples disclosed herein can be effectively scaled across additional guest VMs without the need for additional software and / or hardware development to tailor NVMe drivers for such additional guest VMs.

[0033] The example guest queue managers 216 manage the guest queues 226a, 226b corresponding to the guest VMs 202a, 202b. For example, to access data in the NVMe device 206, the guest VMs 202a, 202b use their guest native NVMe drivers 214a, 214b to generate I / O commands including data access requests (e.g., read and / or write requests). In the illustrated example, the guest queues 226a, 226b are implemented using a ring queue or circular queue. In other examples, any other suitable type of queue can be used. In the illustrated example, the data access requests are based on guest physical memory addresses. That is, because the guest native NVMe drivers 214a, 214b operate as if they are directly interfacing with the NVMe device 206, the guest native NVMe drivers 214a, 214b access data based on guest versions of physical memory addresses (e.g., guest physical memory addresses) that the guest native NVMe drivers 214a, 214b interpret as physical memory addresses of the NVMe device 206, even though the guest physical memory addresses are not actual physical memory addresses of the NVMe device 206.

[0034] In the illustrated example, the guest queue manager 216 receives commands from the guest native NVMe drivers 214a, 214b of the guest VMs 202a, 202b and submits these commands to corresponding guest queues in the guest queues 226a, 226b. The guest queues 226a, 226b of the illustrated example are implemented in memory-mapped input / output (MMIO) registers of the VMM 208. However, any other registers and / or memory spaces can be used. The guest queue manager 216 of the illustrated example also operates as a scheduler to schedule certain ones of the multiple commands in the guest queues 226a, 226b to be serviced by the example meddler 218. The example meddler 218 synchronizes the guest queues 226a, 226b and the shadow queues 230a, 230b so that the host native NVMe driver 222 can provide commands from the shadow queues 230a, 230b to the NVMe device 206. In the illustrated example, to provide commands to the NVMe device 206, the host native NVMe driver 222 synchronizes the shadow queues 230a, 230b with corresponding physical queues 231 in the NVMe device 206. In examples disclosed herein, the meddler 218 can use a snoop technique and / or a polling technique to perform such synchronization. In the illustrated example, the shadow queues 230a, 230b and the physical queues 231 are implemented using ring queues or circular queues. In other examples, any other suitable type of queue can be used.

[0035] In an example snoop technique, the meddler 218 synchronizes the guest queues 226a, 226b and the shadow queues 230a, 230b by snooping submissions from the guest native NVMe drivers 214a, 214b to the guest queues 226a, 226b. In examples where the guest queues 226a, 226b are implemented by MMIO registers, the submissions to the guest queues 226a, 226b are snooped by snooping commands submitted from the guest native NVMe drivers 214a, 214b to the MMIO registers.

[0036] In example polling techniques, the mediator 218 uses a dedicated CPU core / thread to poll the guest queues 226a, 226b for updates. In such examples, commands submitted to the guest queues 226a, 226b (e.g., MIMO registers) are not captured. Instead, the example mediator 218 uses a RAM page to back the guest MMIO register page that implements the guest queues 226a, 226b. The RAM page can be implemented in volatile memory 210 and / or use register space in the NVMe device 206 to implement the RAM page. In this way, when the guest native NVMe drivers 214a, 214b write to (e.g., submit commands to) or read from (e.g., read completion status entries from) the guest queues 226a, 226b, such interactions with the guest queues 226a, 226b are performed directly with the RAM page. The example mediator 218 uses a detection thread to monitor changes in the RAM page and take action in response to detecting changes made by either of the guest native NVMe drivers 214a, 214b.

[0037] When the example mediator 218 captures submitted commands in the guest queues 226a, 226b or obtains submitted commands based on polling, it emulates corresponding access to the physical hardware implemented by the example NVMe device 206 by translating the guest physical memory addresses of the commands in the guest queues 226a, 226b to host physical memory address-based commands. In the illustrated example, the host physical memory addresses are actual physical memory addresses of the NVMe device 206. In the illustrated example, to perform the address translation between guest physical memory addresses and host physical memory addresses, the mediator 218 includes and / or accesses an address translation table (ATT) 228. The example ATT 228 includes a mapping of host physical memory addresses to corresponding guest physical memory addresses. The example shadow queue manager 220 receives the translated commands from the example mediator 218 and places the translated commands in corresponding shadow queues 230a, 230b. The shadow queue manager 220 of the illustrated example also operates as a dispatcher to schedule certain ones of the multiple translated commands in the shadow queues 230a, 226b to be serviced by the host native NVMe driver 222. In some examples, the shadow queues 230a, 230b can be generated directly in the NVMe device 106.

[0038] The example host native NVMe driver 222 accesses certain ones of the plurality of translated commands from the shadow queues 230a, 230b and requests that the NVMe device 206 service the commands. In the illustrated example, the NVMe device 206 includes physical data stores 232a, 232b at separate host memory address ranges. Each physical data store 232a, 232b is allocated as a virtual data store resource to a corresponding one of the guest VMs 202a, 202b. As such, a translated I / O command that includes a data access request corresponding to guest VM-A 202a is handled by the host native NVMe driver 222 by requesting access to data in data store A 232a. Similarly, a translated I / O command that includes a data access request corresponding to guest VM-A 202a is handled by the host native NVMe driver 222 by requesting access to data in data store B 232b.

[0039] In the illustrated example, to improve data access performance, the NVMe device 206 services data access requests from the host native NVMe driver 222 by performing DMA operations 233 to copy the requested data between the corresponding ones of the physical data stores 232a, 232b and the corresponding example guest memory buffers 234a, 234b. In this way, bulk data transfer operations are offloaded from the CPU(s) of the host machine 204. For example, an I / O command that includes such a data access request also includes a physical memory address of the guest memory buffer 234a, 234b to / from which the DMA operation 233 should copy the requested data from / to the NVMe device 206. For example, if the I / O command is a data access request to read data from the NVMe device 206, then the DMA operation 233 copies the data from the NVMe device 206 to the corresponding one of the guest memory buffers 234a, 234b. Alternatively, if the I / O command is a data access request to write data to the NVMe device 206, then the DMA operation 233 copies the data from the corresponding one of the guest memory buffers 234a, 234b to the NVMe device 206.

[0040] In the illustrated example, DMA operation 233 results in a zero CPU cycle copy (zero copy) operation because the bulk data transfer between NVMe device 206 and guest memory buffers 234a, 234b is not handled by the CPU of host machine 204 and, thus, does not place a CPU cycle load on host machine 204. Additionally, the bulk data copy operation performed by DMA operation 233 can be much faster than a copy operation handled by the CPU of host machine 204.

[0041] In the illustrated example, each of guest memory buffers 234a, 234b is allocated as a virtual memory resource to a corresponding one of guest VMs 202a, 202b. In this way, guest VMs 202a, 202b can access requested data from guest memory buffers 234a, 234b. In some examples, subsequent I / O commands from guest VMs 202a, 202b requesting to read and / or write data in NVMe device 206 that has previously been copied to guest memory buffers 234a, 234b are intercepted by virtual NVMe device 224 without being forwarded to shadow queues 230a, 230b, and virtual NVMe device 224 provides requested access to the data in guest memory buffers 234a, 234b instead of re-requesting the data from NVMe device 206. In this way, because read / write speeds to volatile memory are generally faster than read / write speeds to NV memory, access to data that is already resident in guest memory buffers 234a, 234b will be relatively faster than re-requesting the data from NVMe device 206. Additionally, by intercepting such subsequent I / O commands requesting access to data that is already located in guest memory buffers 234a, 234b, virtual NVMe device 224 conserves resources of NVMe device 206 to service other performance critical data access requests. Thus, virtual NVMe device 224 improves performance of data access requests by translating guest I / O commands and submitting the translated I / O commands to shadow queues 230a, 230b when such I / O commands request data in guest memory buffers 234a, 234b that is not available from NVMe device 206, and intercepts guest I / O commands requesting data in guest memory buffers 234a, 234b that is available without requesting it from NVMe device 206.

[0042] In the illustrated example, mediator 218 uses ATT 228 to translate between host physical memory addresses and guest physical memory addresses corresponding to guest memory buffers 234a, 234b, enabling virtual NVMe device 224 to provide guest native NVMe drives 214a, 214b with access to data in the corresponding guest memory buffers 234a, 234b. The dashed lines indicated by reference numeral 242 show shadow queues 230a, 230b corresponding to the respective physical data stores 232a, 232b in NVMe device 206. Additionally, the dashed lines indicated by reference numeral 244 show guest queues 226a, 226b corresponding to the corresponding physical data stores 234a, 234b in volatile memory 210.

[0043] Figure 3 It can be used to achieve combination Figure 2 This describes an example view of the NVMe protocol on a PCIe bus using the example ZCBV-MPT technology. The example view shows example PCI configuration register 302, example command register 304, example management queue 306, and example I / O queue 307. Example PCI configuration register 302, example command register 304, example management queue 306, and example I / O queue 307 can be... Figure 2 Access to the virtual NVMe device 224, in order to communicate with Figure 2 The host native NVMe drive 222 communicates. The illustrated example's PCI configuration register 302 stores and... Figure 2 The base address register (BAR) corresponding to the physical data storage in the NVMe device 206. For example, the BAR0 register 308a stores the lower half of the base memory address of the command register 304 (e.g., the lower 32 bits of a 64-bit long memory address), and the BAR1 register 308b stores the upper half of the base memory address of the command register 304 (e.g., the higher 32 bits of a 64-bit long memory address).

[0044] Implement the example management queue 306 and the example I / O queue 307 using a circular queue or a circular queue. However, any other type of queue can be used instead. Figure 2 The client queues 226a and 226b, shadow queues 230a and 230b, and physical queue 231 include a management queue structurally similar to the example management queue 306, and an I / O queue structurally similar to the example I / O. The illustrated example management queue 306 includes a commit queue 0 (SQ0) (i.e., ASQ / SQ0 312) for the management commit queue (ASQ) and a completion queue (CQ0) (i.e., ACQ / CQ0 314) for the management completion queue (ACQ). Example I / O queue 307 can be implemented...Figure 2 The shadow queues 230a, 230b are similar in structure to the client queues 226a, 226b and are used by the I / O queues 307 to implement the example ZCBV-MPT technology. The I / O queues 307 of the illustrated example include an I / O submission queue 316 and an I / O completion queue 318.

[0045] The example command register 304, management queues 306, and I / O queues 307 are implemented using MMIO registers. However, any other type of register and / or memory space can alternatively be used. The command register 304 of the illustrated example includes an address and / or doorbell (DBL) for submitting commands to the management queues 306 and / or I / O queues 307 to implement the example ZCBV-MPT technology. For example, the command register 304 stores an ASQ memory address 322 at which the ASQ / SQ0 312 begins and an ACQ memory address 324 at which the ASQ / CQ0 314 begins. For the management queues implemented in the client queues 226a, 226b, the ASQ memory address 322 and the ACQ memory address 324 are virtual memory addresses. For the management queues implemented in the shadow queues 230a, 230b and the physical queues 231, the ASQ memory address 322 and the ACQ memory address 324 are physical memory addresses. Although not shown, the command register 304 also stores other information to facilitate other device functionality.

[0046] In the illustrated example, a SQO doorbell (DBL) tail (SQOTDBL) 326 in the control registers 304 stores a tail index value for the ASQ / SQO 312. In examples disclosed herein, the DBL acts as a queue change notification operation to notify that a change has been made to the queue. The virtual NVMe device 224 can write a management command to the tail of the ASQ / SQO 312 based on the SQOTDBL 326. Writing to the tail of the ASQ / SQO 312 submits the management command to the host native NVMe driver 222. In the illustrated example, a CQO doorbell (DBL) head (CQOHDBL) 328 in the control registers 304 stores a head index value for the ASQ / CQO 314. The virtual NVMe device 224 can read the completion status of the management command from the head of the ASQ / CQO 314 based on the CQOHDBL 328. Additionally, the virtual NVMe device 224 writes to the head of the ASQ / CQO 314 to notify the host native NVMe driver 222 that the virtual NVMe device 224 has read the completion status. When implemented in the guest queues 226a, 226b, the SQOTDBL 326 is a virtual tail index value for a guest management submission queue similar to the ASQ / SQO 312, and the CQOHDBL 328 is a virtual head index value for a guest management completion queue similar to the ASQ / CQO 314. When implemented in the shadow queues 230a, 230b and the physical queues 231, the SQOTDBL 326 is a physical tail index value for a shadow or physical management submission queue similar to the ASQ / SQO 312, and the CQOHDBL 328 is a physical head index value for a shadow or physical management completion queue similar to the ASQ / CQO 314.

[0047] In the illustrated example, the SQ1 Doorbell (DBL) Tail (SQ1TDBL) 330 in control register 304 stores the tail index value of the I / O submission queue 316. The virtual NVMe device 224 can write I / O commands (e.g., data access requests) to the tail of the I / O submission queue 316 based on the SQ1TDBL 330. Writing to the tail of the I / O submission queue 316 submits the I / O command to the host native NVMe driver 222. In the illustrated example, the CQ1 Doorbell (DBL) Head (CQ1HDBL) index value 332 in control register 304 stores the head memory address of the I / O completion queue 318. The virtual NVMe device 224 can read the completion status of I / O commands from the head of the I / O completion queue 318 based on the CQ1HDBL memory address 332. Additionally, the virtual NVMe device 224 writes to the header of the I / O completion queue 318 to notify the host native NVMe driver 222 that the virtual NVMe device 224 has completed reading. When implemented in client queues 226a and 226b, SQ1TDBL 330 is a virtual tail index value of the client I / O submission queue, similar to I / O submission queue 316, and CQ1HDBL 332 is a virtual head index value of the client I / O completion queue, similar to I / O completion queue 318. When implemented in shadow queues 230a and 230b and physical queue 231, SQ1TDBL 330 is a physical tail index value of the shadow or physical I / O submission queue, similar to I / O submission queue 316, and CQ1HDBL 332 is a physical head index value of the shadow or physical I / O completion queue, similar to I / O completion queue 318.

[0048] Figure 4 This shows the combination of the above text. Figure 2 The example ZCBV-MPT technology is used to facilitate DMA data transfers (e.g., zero-copy operations). Figure 2 Example mediator 218 and example virtual NVMe device 224. Figure 4 In the illustrated example, a representative view of the guest VM-A 202a is shown using the guest native NVMe drive 214a and the corresponding guest queue 226a accessed by the guest native NVMe drive 214a. Figure 4 The illustrated example also shows a corresponding shadow queue 230a. The shadow queue 230a in the illustrated example is mapped to the NVMe device 206 (…). Figure 2 Physical queue 231 in ). For example, when shadow queue manager 220 makes changes to shadow queue 230a, host native NVMe drive 222 ( Figure 2 This will change the propagation or synchronization to physical queue 231.

[0049] In the illustrated example, the guest queues 226a, the shadow queues 230a, and the physical queues 231 include management queues (e.g., ASQ, ACQ) and I / O queues (IOSQ, IOCQ). The management queues are used for management commands that manage the virtual NVMe device 224, manage queues, get / set driver configuration information, etc. The I / O queues are used for I / O commands such as data access requests that access data in the NVMe device 206 Figure 2 ) (e.g., by the host machine 204 Figure 2 ) hosting all VMs. The size of a physical management queue can be different from the size of its corresponding guest management queue. The I / O queues are statically divided into multiple groups, and each group is assigned to a corresponding VM for exclusive use. In addition, one I / O queue of the physical queues 231 uniquely corresponds to one shadow I / O queue of the shadow queues 230a and one guest I / O queue of the guest queues 226a in a one-to-one manner. In addition, the size of a physical I / O queue and its corresponding shadow queue and guest I / O queue are the same.

[0050] The management queues are used to manage the NVMe device 206. For example, if the guest VM 202a wants to use the virtual NVMe device 224, the guest VM 202a sends a message to a management queue (e.g., the ASQ of the guest queue 226a) to get the capabilities of the virtual NVMe device 224. In the examples disclosed herein, to save resources of the underlying physical hardware (e.g., the NVMe device 206) to better handle performance critical I / O commands, the guest queue manager 216 Figure 2) processing management commands in the guest queue 226a to determine which management commands need to be forwarded to the shadow queue 230a (and, thus, the physical queue 231) and which management commands can be intercepted and processed by the virtual NVMe device 224 without being forwarded to the shadow queue 230a. This determination is based on the type of management command. For example, two types of management commands are mandatory commands and optional commands (e.g., virtual asynchronous events). Optional commands or virtual asynchronous events have no impact on the physical device (e.g., they are not intended to be completed by the NVMe device 206) and, thus, are not forwarded from the guest queue 226a to the shadow queue 230a and the physical queue 231. An example of an optional command is an “identify” command that the guest VM 202a can use to request the identity of the virtual NVMe device 224. For commands that do have an impact on the physical device operation (e.g., access (change / request) configuration of the physical device that the virtual NVMe device 224 is not aware of), the guest queue manager 216 forwards these commands to the shadow queue 230a and the management queue (e.g., ASQ) of the physical queue 231. However, before forwarding the management command, the mediator 218 performs a translation as described below in connection with FIG. 3 to ensure that the management command is safe (e.g., to ensure that the management command does not interfere with commands from another guest queue of another VM (such as the guest VM 202b) in the management queue of the physical queue 231. Figure 5 and Figure 6 For example, if the management command is a “delete I / O queue,” the mediator 218 confirms whether the I / O queue to be deleted belongs to the guest queue 226a that sent the delete command. If not, it is possible that the I / O queue of another guest VM is deleted. As such, the mediator 218 intercepts the “delete I / O queue” management command without forwarding it to the shadow queue 230a and the physical queue 231. Figure 2

[0051] ​In the illustrated example, the mediator 218 translates commands from the guest queue 226a and copies the translated commands to the shadow queue 230a. To translate commands from the guest queue 226a to the shadow queue 230a, the mediator 218 translates virtual parameters (e.g., virtual memory addresses of data to be accessed, virtual queue identifiers of guest queues 226a, 226b used by virtualized resources such as guest VMs 202a, 202b, etc.) to physical parameters (e.g., physical memory addresses of data to be accessed, physical queue identifiers of shadow queues 230a, 230b and / or physical queues 231 used by physical resources such as NVMe device 206, etc.). In the illustrated example, the mediator 218 translates guest physical memory addresses (GPAs) to host physical memory addresses (HPAs). Example GPAs are emulated physical memory addresses corresponding to the virtual NVMe device 224, such that the virtual NVMe device 224 operates as if it were an actual physical device. The example GPAs are used as emulated physical memory addresses of data to be accessed when the guest native NVMe driver 214a specifies data to be accessed in the NVMe device 206. The example HPAs are used by the host native NVMe driver 222 to specify actual physical locations of data in the NVMe device 206. For example, the mediator 218 can translate commands from the guest queue 226a by performing an example virtual physical memory address to guest physical memory address translation and an example guest physical memory address to host physical memory address translation. The example virtual physical memory address to guest physical memory address translation involves the mediator 218 translating a virtual memory address of data to be accessed to a corresponding GPA corresponding to the virtual NVMe device 224. The example guest physical memory address to host physical memory address translation involves the mediator 218 translating the GPA to a corresponding HPA for use by the NVMe device 206. The example mediator 218 also translates guest logical block addresses (GLBAs) to host logical block addresses (HLBAs). The guest native NVMe driver 214a uses GLBAs to specify logical addresses of data. The host native NVMe driver 222 uses HLBA to specify logical addresses of data. The example mediator 218 also translates guest queue identifiers (GQIDs) (e.g., virtual queue identifiers of guest queues 226a, 226b) to host queue identifiers (HQIDs) (e.g., physical queue identifiers of shadow queues 230a, 230b and / or physical queues 231). The guest native NVMe driver 214a uses GQIDs to specify the guest queue 226a. The host native NVMe driver 222 uses HLBA to specify the shadow queue 232a. The mediator 218 can also perform one or more additional or alternative translations of parameters.In the illustrated example, mediator 218 and shadow queue manager 220 work together to create shadow queue 230a to submit new, transformed commands to NVMe device 206. Figure 4 In the example shown, as in conjunction with the above... Figure 2 The process of handling translated I / O commands (e.g., translated data requests) in the shadow queue 230a by the host native NVMe driver 222 enables... Figure 2 The NVMe device 206 performs a DMA operation 233 (e.g., a zero-copy operation) to copy data between the NVMe device 206 and the guest memory buffer 234a of the requesting guest VM 202a.

[0052] Figure 5 This demonstrates emulated PCI configuration and management. Figure 2 The example guest VM-A 202a guest queue 226a is used to implement the example ZCBV-MPT technology. Figure 2 Example virtual NVMe device 224. In Figure 5 In the illustrated example, the virtual NVMe device 224 manages the client PCI configuration 502 and the client command register 504. The client PCI configuration 502 in the illustrated example is structurally and operationally similar to... Figure 3 The PCI configuration is similar to 302. However, in Figure 5 In the illustrated example, guest PCI configuration 502 is emulated by virtual NVMe device 224 to serve as a virtual PCI interface for guest native NVMe driver 214a of guest VM 202a. For example, guest PCI configuration 502 includes a base address register (BAR) that is interpreted by guest native NVMe driver 214a as command register 304. In this manner, requests for access to the PCI bus made by guest native NVMe driver 214a are captured by virtual NVMe device 224, which uses guest PCI configuration 502 to emulate guest native NVMe driver 214a's access to the PCI bus.

[0053] The client command register 504 is structurally and operationally consistent with the above. Figure 3 The command register 304 is similar. However, the virtual NVMe device 224 emulates the client command register 504 for use by the client queue manager 216 and the client native NVMe driver 214a to access the client queue 226a. In this way, commands written to the client queue 226a are captured by the virtual NVMe device 224 to emulate access to the underlying physical resources (such as...). Figure 2Access to the NVMe device 206). In the illustrated example, mediator 218 dispatches translated commands from client queue 226a to shadow queue 230a. This in Figure 5 In the illustrated example, mediator 218 sends a micro-operation (Qop) notification 508 to shadow queue manager 220. In this way, host native NVMe drive 222 can service commands from shadow queue 230a. In the illustrated example, host native NVMe drive 222 uses host command register 510 to identify the location of the memory mapping of physical queue 231, enabling NVMe drive 222 and NVMe device 206 to service commands synchronized to physical queue 231.

[0054] When the host native NVMe drive 222 completes a command, it writes the completion message to the shadow queue 230a. In this manner, the shadow queue manager 220 sends a DBL notification 514 to the mediator 218 in response to the completion message being written to the shadow queue 230a. The example mediator 218 transforms the completion queue entry from the shadow queue 230a and writes the transformed completion queue entry to the guest queue 226a. The example guest native NVMe drive 214a then accesses the transformed completion queue entry from the guest queue 226a. For example, the completion queue entry might indicate to the guest native NVMe drive 214a that data requested from the NVMe device 206 is stored corresponding to the guest VM 202a. Figure 2 In the memory buffer 234a.

[0055] Figure 6 It shows the submission to Figure 2 The example client queue 226a manages I / O commands (e.g., data access requests). Figure 2 Example shadow queue 230a and example client queue 226a are used to implement the example ZCBV-MPT technology disclosed herein. Figure 2 Example virtual NVMe device 224. Although described in conjunction with handling I / O commands. Figure 6 This is an example, but similar operations can be used to handle management commands. Figure 6 The example can be used to serve I / O commands written by the guest VM 202a to the guest queue 226a (e.g., requests to access). Figure 2 (Data in NVMe device 206). Figure 6 The example illustrates multiple blocks representing operations performed by the virtual NVMe device 224. The example blocks represent operations that can be performed by one or more processors (e.g., ...). Figure 13by the one or more processors 1312) to implement corresponding operations. In the illustrated example, the client queue manager 216 Figure 2 captures changes to a submission queue DBL (SQDBL) entry (block 602). For example, the SQDBL entry is used as a notification by the client native NVMe driver 214a that an I / O command has been added to the client queue 226a.

[0056] The example client queue manager 216 copies the I / O command from an I / O submission queue (IOSQ) of the client queue 226a (block 604). The example mediator 218 Figure 2 parses the I / O command (block 606). For example, the mediator 218 identifies address portions of the I / O command including the GPA, GLBA, GQID, etc. The example mediator 218 translates the I / O command (block 608). For example, the mediator 218 translates the GPA to an HPA, the GLBA to an HLBA, the GQID to an HQID, etc. The shadow queue manager 220 Figure 2 writes the translated I / O command to the shadow queue 230a (block 610). For example, the shadow queue manager 220 writes the translated I / O command to the corresponding IOSQ identified by the HQID of the shadow queue 230a. The client queue manager 216 modifies a DBL register value of a corresponding one of the plurality of client queues 226a (block 612). For example, the client queue manager 216 modifies the DBL register value corresponding to the IOSQ of the client queue 226a to acknowledge that the I / O command located therein has been synchronized with the shadow queue 230a. The shadow queue manager 220 modifies a DBL register value of a corresponding one of the plurality of physical queues 231 (block 614). For example, the shadow queue manager 220 modifies the DBL register value corresponding to the IOSQ of the physical queue 231 to acknowledge that the I / O command located therein has been synchronized with the client queue 230a.

[0057] Figure 7 management of completion status entries based on I / O commands (e.g., data access requests) submitted to the shadow queue 230a that indicate completion is shown Figure 2 example shadow queue 230a and example client queue 226a to implement the example ZCBV-MPT techniques disclosed herein Figure 2 example virtual NVMe device 224. Although examples are described in connection with processing I / O commands, similar operations can be used to process management commands. After the example process described above in connection with Figure 7 Figure 6 described above in connection with processing I / O commands, the example process described above in connection with Figure 7 ​of the I / O command. For example, if the I / O command is a request to access data (e.g., read / write data) in the NVMe device 206, the completion status indicates the result of the I / O command. Figure 2 Examples of the I / O command completion status 232a are shown in FIG. 7. For example, if the I / O command is a request to access data (e.g., read / write data) in the NVMe device 206, the completion status indicates the result of the I / O command. Figure 7 Examples of the I / O command completion status 232a are shown in FIG. 7. For example, if the I / O command is a request to access data (e.g., read / write data) in the NVMe device 206, the completion status indicates the result of the I / O command. Figure 2 Examples of the I / O command completion status 232a are shown in FIG. 7. For example, if the I / O command is a request to access data (e.g., read / write data) in the NVMe device 206, the completion status indicates the result of the I / O command. Figure 7 Examples of the I / O command completion status 232a are shown in FIG. 7. For example, if the I / O command is a request to access data (e.g., read / write data) in the NVMe device 206, the completion status indicates the result of the I / O command. Figure 13 Examples of the I / O command completion status 232a are shown in FIG. 7. For example, if the I / O command is a request to access data (e.g., read / write data) in the NVMe device 206, the completion status indicates the result of the I / O command.

[0058] In the illustrated example, after completing the example DMA operation 233( Figure 2 and Figure 4 ), the shadow queue manager 220( Figure 2 ) detects an interrupt in response to the host native NVMe driver 222 submitting a completion status to the IOCQ of the shadow queue 230a (block 702). For example, the completion status is generated by the host native NVMe driver 222 to indicate that the I / O command has been serviced and completed. The example mediator 218( Figure 2 ) parses the completion status entry (block 704). For example, the mediator 218 identifies the address portion of the completion status entry including the GPA, GLBA, GQID, etc. The example mediator 218 translates the completion status entry (block 706). For example, the mediator 218 translates the HPA to GPA, the HLBA to GLBA, the HQID to GQID, etc. The guest queue manager 216( Figure 2The transformed completion status entry is written to the IOCQ of client queue 226a (box 708). For example, client queue manager 216 writes the transformed completion status entry to the corresponding IOCQ identified by the GQID of client queue 226a. Client queue manager 216 modifies the DBL register value of a corresponding client queue among the plurality of client queues 226a (box 710). For example, client queue manager 216 modifies the DBL register value corresponding to the IOCQ of client queue 226a to confirm that the completion status entry therein has been synchronized with shadow queue 230a. Shadow queue manager 220 modifies the DBL register value of a corresponding physical queue among the plurality of physical queues 231 (box 712). For example, shadow queue manager 220 modifies the DBL register value corresponding to the IOCQ of physical queue 231 to confirm that the completion status entry therein has been synchronized with client queue 226a. In the illustrated example, client queue manager 216 asserts a client interrupt (box 714). For example, if interrupts are enabled for guest VM 202a, guest queue manager 216 uses such guest interrupts to notify guest native NVMe drive 214a I / O commands that have been completed.

[0059] Figure 8 The executable is shown to define Figure 2 and Figures 4-6 The virtual NVMe device 224 interfaces to implement example machine-readable instructions for the example ZCBV-MPT technology disclosed herein. Figure 8 In the illustrated example, the physical resource definition section 802 defines the NVMe device 206 to be allocated to the virtual NVMe device 224. Figure 2 The physical parameters of a physical resource definition (LBA). For example, the physical resource definition section 802 defines the start of the physical LBA (e.g., the start of the HLBA), the number of sectors, and the sector size. Similarly, in... Figure 8 In the example, the queue mapping section 804 defines the mapping between shadow queues 230a and 230b and the corresponding physical queues of physical queue 231. Figure 2 and Figures 4-7 ).

[0060] Figure 9 The executable is shown to define Figure 2 and Figures 4-6 The virtual NVMe device 224 is configured to implement example machine-readable instructions of the example ZCBV-MPT technology disclosed herein. For example, these functions include a shadow completion queue creation function 902 for the virtual NVMe device 224 to execute in physical queue 231 (…). Figure 2 and Figures 4-7 NVMe devices in 206 ( Figure 2) to create shadow completion queues (e.g., IOCQ or ACQ) when generating completion status entries. Example functionality also includes shadow submission queue creation functionality 904 for use by the virtual NVMe device 224 in creating shadow submission queues (e.g., IOSQ or ASQ) when submitting commands (e.g., I / O commands or management commands) to the shadow queues 226a, 226b. Figure 2 ) to create shadow completion queues (e.g., IOCQ or ACQ) when generating completion status entries. Example functionality also includes shadow submission queue creation functionality 904 for use by the virtual NVMe device 224 in creating shadow submission queues (e.g., IOSQ or ASQ) when submitting commands (e.g., I / O commands or management commands) to the shadow queues 226a, 226b. Figure 2 ) to create shadow completion queues (e.g., IOCQ or ACQ) when generating completion status entries. Example functionality also includes shadow submission queue creation functionality 904 for use by the virtual NVMe device 224 in creating shadow submission queues (e.g., IOSQ or ASQ) when submitting commands (e.g., I / O commands or management commands) to the shadow queues 226a, 226b.

[0061] Figure 10 Figure illustrates a host machine 1002 implementing example ZCBV-PVIO techniques to provide VMs (e.g., guest VMs 1004) with access to physical NV storage. In the illustrated example, the host machine 1002 executes an example VMM 1006, which can be implemented using a host Linux / KVM OS or any other suitable host OS or hypervisor. However, the example ZCBV-PVIO techniques disclosed herein bypass the VMM 1006 to provide faster access to the NVMe device 106 than is achieved using existing virtualization techniques that access NV storage. Figure 10 The example ZCBV-PVIO techniques involve executing a PVIO FE block driver 1008 in the guest VMs 1002 and a BE block service driver 1012 in the IOVM 1014 to implement zero-copy data transfers using DMA operations 1018 between the NVMe device 206 and guest memory 1022 located in volatile memory 1024.

[0062] In Figure 10In the illustrated example, the PVIO FE block driver 1008 can be implemented using any suitable PVIO FE block driver having an interface optimized with virtualization. The example PVIO FE block driver 1008 uses a shared ring buffer 1026 (or circular buffer) to communicate between the guest VM 1002 and the IOVM 1014. However, any other type of buffer can alternatively be used. In the illustrated example, the shared ring buffer 1026 is created in system memory (e.g., system memory of the host machine 1002). However, in other examples, the shared ring buffer can be created in any other memory. The shared ring buffer 1026 of the illustrated example includes memory address spaces of I / O operation descriptors to which the DMA operations 1018 are to copy data from the NVMe device 206 (e.g., perform a bulk data transfer) to a corresponding one of the guest memory buffers 1022 / from a corresponding one of the guest memory buffers 1022 to the NVMe device 206.

[0063] In Figure 10In the illustrated example, the BE block service driver 1012 receives a virtual interrupt request (IRQ) notification from the shared ring buffer 1026 indicating that the PVIO FE block driver 1008 has submitted an I / O request to the shared ring buffer 1026. In other examples, instead of using a virtual IRQ notification, the PVIO FE block driver 1008 can poll the shared ring buffer 1026 for new I / O requests. In the illustrated example, the example buffer interface 1042 of the example BE block service driver 1012 accesses the I / O request in the shared ring buffer 1026, and the example queue interface 1044 of the BE block service driver 1012 works with the example native NVMe driver 1032 executed by the IOVM 1014 to create an I / O queue 1034 for the I / O request. In some examples, the queue interface 1044 can create multiple I / O queues 1034 simultaneously to service multiple I / O requests from the guest VM 1002. The example I / O queue 1034 can be implemented using a ring queue or a circular queue. However, any other type of queue can alternatively be used. The example I / O queue 1034 can be created in system memory and / or any other suitable memory of the host machine 1002. In the illustrated example, the example translator 1046 of the BE block service driver 1012 translates virtual parameters (e.g., guest parameters) of the I / O request to physical parameters (e.g., host parameters). For example, the example translator 1046 can translate virtual memory addresses that map to physical locations in the NVMe device 206 to physical memory addresses of those physical locations in the NVMe device 206. In the illustrated example, the I / O request includes a DMA descriptor submitted in the I / O queue 1034 to identify a host physical address of the corresponding guest memory buffer 1022. In this way, the I / O queue 1034 submits the I / O request and its DMA descriptor to the NVMe device 206 so that the NVMe device 206 can use the DMA descriptor to perform a DMA operation 1018 for a bulk data transfer of the requested data by directly accessing the host physical address of the corresponding guest memory buffer 1022. After the DMA operation 1018, the example notifier 1048 of the BE block service driver 1012 notifies the guest VM 1002 of completion of the I / O request.

[0064] By performing the DMA operation 1018, the NVMe device 206 directly accesses the guest memory buffer 1022, bypassing interception of the bulk data transfer between the NVMe device 206 and the guest memory buffer 1022 by the VMM 1006. This is in contrast to the example of FIG. 1, in which the VMM 1006 intercepts the bulk data transfer between the NVMe device 106 and the guest memory buffer 102, and performs the DMA operation 108 using the DMA descriptor 110. Figure 10The communication 1038 between the VMM bypass I / O queue 1034 and the guest memory buffer 1022 and between the shared ring buffer 1026 and the guest memory buffer 1022 is shown in the illustrated example by the dashed lines representing the communication 1038. For example, from the perspective of the guest VM 1002 and the shared ring buffer 1026, Figure 10 The example ZCBV-PVIO technique results in a virtual DMA operation 1042 because the data can be accessed quickly using the guest memory buffer 1022 without the need for a lengthy data transfer process via the VMM 1006.

[0065] While the example ZCBV-MPT technique is disclosed in conjunction with Figures 2-9 the example ZCBV-PVIO technique is disclosed in conjunction with Figure 10 one or more of the elements, processes and / or devices illustrated in Figures 2-10 may be combined, divided, re-arranged, omitted, eliminated and / or implemented in any other way. Further, the example guest native drivers 214a, 214b Figure 2 , the example virtual NVMe device 224 Figure 2 , the example guest queue manager 216 Figure 2 , the example mediator 218 Figure 2 , the example shadow queue manager 220 Figure 2 , the example host native NVMe driver 222 Figure 2 , the example ATT 228 Figure 2 , the example PVIO FE block driver 1008 Figure 10 , the example BE block service driver 1012 Figure 10 , the example buffer interface 1042 Figure 10 , the example queue interface 1044 Figure 10 , the example translator 1046 Figure 10 , the example notifier 1048 Figure 10 , and / or the example native NVMe driver 1032 Figure 10 may be implemented by hardware, software, firmware or any combination of hardware, software, and / or firmware. Thus, for example, the example guest native drivers 214a, 214b Figure 2 , the example virtual NVMe device 224 Figure 2 , the example guest queue manager 216 Figure 2 , the example mediator 218 Figure 2 , the example shadow queue manager 220 Figure 2 , the example host native NVMe driver 222 Figure 2), example ATT 228 Figure 2 ), example PVIO FE block driver 1008 Figure 10 ), example BE block service driver 1012 Figure 10 ), example buffer interface 1042 Figure 10 ), example queue interface 1044 Figure 10 ), example translator 1046 Figure 10 ), example notifier 1048 Figure 10 ), and / or example native NVMe driver 1032 Figure 10 ) can be implemented by one or more analog or digital circuits, logic circuits, programmable processor(s), application specific integrated circuit(s) (ASICs), programmable logic device(s) (PLD(s)), and / or field programmable logic device(s) (FPLD(s)). When reading any of the apparatus or system claims to cover a purely software and / or firmware implementation, at least one of the example guest native drivers 214a, 214b Figure 2-10 ), example virtual NVMe device 224 Figures 2-10 ), example guest queue manager 216 Figure 2 ), example mediator 218 Figure 2 ), example shadow queue manager 220 Figure 10 ), example host native NVMe driver 222 Figure 10 ), example ATT 228 Figure 2 ), example PVIO FE block driver 1008 Figure 10 ), example BE block service driver 1012 Figure 2 ), example buffer interface 1042 Figure 10 ), example queue interface 1044 Figure 10 ), example translator 1046 Figure 2 ), example notifier 1048 Figure 10 ), and / or example native NVMe driver 1032 Figure 10 ) is hereby expressly defined to include a non-transitory computer readable storage device or storage disk (such as memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc.) including the software and / or firmware. Yet further, example ZCBV-MPT technology and / or ZCBV-PVIO technology can include one or more elements, processes, and / or devices in addition to or instead of the elements, processes, and / or devices shown in Figure 2 Figure 2 ​​

[0066] In the example disclosed in this article, access is submitted to Figure 10 The command access device for the commands in client queues 226a and 226b can be provided by Figure 10 This is implemented using the client queue manager 216. Additionally, access is submitted to... Figure 2 The example command access device for the shared ring buffer 1026 can be provided by... Figure 11 The buffer interface 1042 is used for implementation. In the examples disclosed herein, the conversion device for generating the converted commands can be provided by... Figures 2-9 The regulator 218 and / or Figure 12 The converter 1046 is used to implement this. In the example disclosed herein, the converted command is submitted to... Figure 10 The command submission device for the shadow queues 230a and 230b can be implemented by the mediator 218. Additionally, it is used to submit the converted commands to... Figure 13 The example command submission device of I / O queue 1034 can be used Figure 11 This is implemented using the queue interface 1044. In the example disclosed herein, it is used to submit completion status entries to... Figure 12 The completion submission mechanism for client queues 226a and 226b can be implemented by client queue manager 216. Additionally, a mechanism for submitting completion status entries to... Figure 11 The example completion submission device of the shared ring buffer 1026 can be provided by Figure 12 The buffer interface 1042 is used for implementation. In the examples disclosed herein, the queue creation device can be implemented by... Figure 2 The client queue manager 216 is used to create client queues 226a, 226b, which can be implemented by... Figure 11 The shadow queue manager 220 and / or mediator 218 are used to create shadow queues 230a, 230b, and / or can be implemented by... Figure 11 queue interface 1042 and / or Figure 11 The native NVMe driver 1032 is used to create I / O queues 1034. In the examples disclosed herein, a command interception device (e.g., ...) is used to determine whether a command should be processed by physical resources. Figure 2 The NVMe device 206 can be implemented by the guest queue manager 216. In the illustrated example, the completion status notification device for notifying the guest VM of the completion of a command submitted by the guest VM can be implemented by... Figure 2 This is achieved through the 1048 notification device.

[0067] Figure 2 The diagram shows the representation used for implementation. Figure 2 A flowchart of example machine-readable instructions for ZCBV-MPT technology, andFigure 2 a flowchart showing example machine-readable instructions implementing the ZCBV-PVIO technique of Figure 2 In these examples, the machine-readable instructions implement a program for execution by one or more processors, such as the processor(s) 1312 shown in the example processor platform 1300 discussed below in connection with Figure 4 The program can be embodied in software stored on a non-transitory computer readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a digital versatile disk (DVD), a Blu-ray disk, or a memory associated with the processor 1312, but the entire program and / or portions of it can alternatively be embodied in firmware or dedicated hardware, and / or be executed by an apparatus other than the processor 1312. In addition, although the flowcharts illustrate the example program, many other methods of implementing the example ZCBV-MPT technique and / or the example ZCBV-PVIO technique can alternatively be used. For example, the order of execution of the blocks can be changed, and / or some of the blocks described can be changed, eliminated, or combined. Additionally or alternatively, any of the blocks or all of the blocks can be implemented by one or more hardware circuits structured to perform the corresponding operations without executing software or firmware. Figure 2 and Figure 4 In these examples, the machine-readable instructions implement a program for execution by one or more processors, such as the processor(s) 1312 shown in the example processor platform 1300 discussed below in connection with

[0068] As mentioned above, the example ZCBV-MPT technique and / or the example ZCBV-PVIO technique can be implemented using encoded instructions (e.g., computer and / or machine readable instructions) stored on a non-transitory computer and / or machine readable medium. Figure 2 and Figure 2example, a hard disk drive, a flash memory, a read-only memory, a compact diskette, a digital versatile disk, a cache, a random access memory, and / or any other storage devices or storage disks in which information is stored for any duration (e.g., for extended periods of time, permanently, for brief instances, for temporary buffering, and / or for caching of the information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and / or storage disk and to exclude propagating signals and transmission media. As used herein, “including” and “comprising” (and any form of these terms, such as “include” and “comprise”) are used as open terms. Therefore, an item, region, or item of matter, comprising any of the recited elements, can include those elements irrespective of whether additional, different, or other elements are also present. As used herein, when the phrase “at least” is used as the transition term in a preamble of a claim, it is used in the same open-ended fashion as the term “comprising” and “including” and the inclusion and involvement of additional, different or other elements are not precluded.

[0069] In conjunction with Figure 2 Client VM-A 202a is described Figure 4 However, Examples of Figure 11 may be implemented simultaneously using Client VM-B 202b, any other client VM, and / or multiple client VMs. Figure 12 The procedure of Figure 12 ) determines whether a command has been submitted. For example, the client queue manager 216 determines whether an I / O command or a management command has been submitted to the client queue 226a (e.g., based on a queue change notification from the client queue 226a). Figure 12 In the illustrated example, if the command has not been submitted, control remains at block 1102 waiting for submission of the command.

[0070] If the command has been submitted to the client queue 226a (block 1104), the example client queue manager 216 accesses the command in the client queue 226a (block 1104). For example, the client queue manager 216 can access an I / O command in an IOSQ of the client queue 226a, or can access a management command in an ASQ of the client queue 226a. The example client queue manager 216 determines whether the command is to be submitted to the shadow queue 230a (block 1106). Figure 10)(block 1106). For example, the client queue manager 216 can determine whether to submit the command to the shadow queue 230a based on whether the command is handled by the NVMe device 206. The client queue manager 216 can make such a determination based on, for example, whether the command is an I / O command requesting access to data in the NVMe device 206 (e.g., reading / writing data) that is not available in the corresponding client memory buffer 234a, or whether the command is a management command for accessing a configuration of the NVMe device 206 (e.g., reading / setting configuration information of the NVMe device 206 that is not available for access in the virtual NVMe device 224). If the client queue manager 216 determines that the command should not be submitted to the shadow queue 230a, control proceeds to block 1108, where the virtual NVMe device 224 services the command. For example, the virtual NVMe device 224 can provide the requested configuration information to the client VM 202a and / or direct the client VM 202a to a location in the client memory buffer 234a that stores data requested by the client VM 202a. Figure 12

[0071] If the client queue manager 216 determines at block 1106 that the command should not be submitted to the shadow queue 230a, the example mediator 218 (block 1110) generates a translated command. For example, the mediator 218 generates the translated command by translating one or more virtual parameters associated with the command of the client VM 202a to one or more physical parameters associated with the NVMe device 206. Example virtual parameters can include a virtual memory address of data to be accessed by the virtualized resource (such as the client VM 202a), a virtual queue identifier of the client queue 226a, 226b, etc. Physical parameters can include a physical memory address of data to be accessed by the physical resource (such as the NVMe device 206), a physical queue identifier of the shadow queue 230a, 230b and / or the physical queue 231, etc. In the illustrated example, if the command is an I / O command, the address of the client memory buffer 234a (block 1112) used to perform the DMA operation 233 (block 1114) is the same in both the original command submitted to the client queue 226a by the client native NVMe driver 214a and the translated command. In this way, data corresponding to the I / O command is accessible to the client VM 202a and the NVMe device 206 at the same client memory buffer 234a. Figure 12 Figure 10 Figure 10 Figure 10 Figure 10

[0072] ​​​​​​The example mediator 218 and / or the shadow queue manager 220 submits the translated command to the shadow queue 230a (block 1112). For example, the mediator 218 and / or the shadow queue manager 220 can submit the translated I / O command to the IOSQ of the shadow queue 230a, or the translated management command to the ASQ of the shadow queue 230a. In some examples, the mediator 218 and / or the shadow queue manager 220 creates the shadow queue 230a prior to submitting the translated command to the shadow queue 230a. Figure 10 ) creates the shadow queue 230a prior to submitting the translated command to the shadow queue 230a.

[0073] The example shadow queue manager 220 determines whether the translated command has been serviced (block 1114). For example, the shadow queue manager 220 can detect an interrupt asserted by the shadow queue 230a in response to the host native NVMe driver 222 Figure 10 ) submits a completion status entry to the IOCQ or ACQ of the shadow queue 230a. In the illustrated example, if the translated command has not been serviced, the shadow queue manager 220 waits for the service of the translated command to complete. When the translated command has been serviced, then control proceeds to block 1116, where the example mediator 218 translates the completion status entry (block 1116). For example, the mediator 218 accesses the completion status entry from the IOCQ or ACQ of the shadow queue 230a, and it translates the completion status entry for use by the guest VM 202a by translating one or more physical parameters to one or more corresponding virtual parameters. In some examples, the completion status entry indicates completion of a DMA operation to copy data from the NVMe device 206 to a guest memory buffer 234a corresponding to the guest VM 202a / copy data from a guest memory buffer 234a corresponding to the guest VM 202a to the NVMe device 206 (e.g., DMA operation 233 of Figure 10 and Figure 10 The example guest queue manager 216 submits the translated completion status entry to the guest queue 226a (block 1118). For example, the guest queue manager 216 writes the translated completion status entry to the IOCQ or ACQ of the guest queue 226a. Figure 10 The example process of FIG. 10 ends.

[0074] Figure 10 is a flowchart representative of example machine readable instructions that can be executed to implement the example ZCBV-PVIO techniques disclosed herein. Although the example routine of Figure 12 is described in connection with I / O commands, the example routine of Figure 13 may be similarly used to process management commands using the example ZCBV-PVIO techniques disclosed herein. Furthermore, although the example routine of FIG. 10 is described in connection with a single guest VM (e.g.,Figures 6-9 the client VM 1002) is described Figure 11 example program that can be implemented to service commands for multiple client VMs concurrently. Figure 12 The program of FIG. 12 begins at block 1202 where the example BE block service driver 1012 Figure 2 ) determines whether a command has been submitted. For example, the BE block service driver 1012 determines whether the PVIO FE block driver 1008 of the client VM 1002 has submitted a command via the shared ring buffer 1026 (block 1202) based on, for example, buffer change notifications from the shared ring buffer 1026 and / or notifications from the PVIO FE block driver 1008 of the submitted command. In the illustrated example, if the command has not been submitted, control remains at block 1202 waiting for submission of the command. Figure 2

[0075] If the command has been submitted to the shared ring buffer 1026 (block 1202), the example buffer interface 1042 Figure 2 ) of the example BE block service driver 1012 accesses the command in the shared ring buffer 1026 (block 1204). The example queue interface 1044 Figure 2 ) of the example BE block service driver 1012 determines whether an I / O queue 1034 Figure 2 ) has been created (block 1206) to submit the command to the native NVMe driver 1032 Figure 2 ). If the example queue interface 1044 determines at block 1206 that the I / O queue 1034 has not been created, control proceeds to block 1208 where the queue interface 1044 and / or the native NVMe driver 1032 creates the I / O queue 1034. For example, the example queue interface 1044 can send a request to the native NVMe driver 1032 to create the I / O queue 1034.

[0076] If the example queue interface 1044 determines at block 1206 that the I / O queue 1034 has been created, the example converter 1046 Figure 2 ​) generating a translated command (block 1210). For example, the translator 1046 generates a translated command by translating one or more virtual parameters associated with the command of the guest VM 1002 to one or more physical parameters associated with the NVMe device 206. Example virtual parameters can include virtual memory addresses used by virtualized resources (such as the guest VM 1002) that map to physical locations in the NVMe device 206 where data is to be accessed, virtual queue identifiers, shared ring buffer identifiers, and the like. Physical parameters can include physical memory addresses of physical locations in the NVMe device 206 where data is located, physical queue identifiers, and the like. For example, the physical parameters are used by physical resources (such as the NVMe device) to service the translated command. In the illustrated example, the address of the guest memory buffer 1022 Figure 10 Figure 10

[0077] The example queue interface 1044 submits the translated command to the I / O queue 1034 (block 1212). For example, the queue interface 1044 can submit the translated I / O command to an IOSQ of the I / O queue 1034.

[0078] ​​Example BE block service driver 1012 determines whether a translated command has been serviced (box 1214). For example, BE block service driver 1012 may detect an interrupt asserted by native NVMe driver 1032 in response to a signal from NVMe device 206 indicating completion of a translated command and / or in response to a completion status entry submitted to I / O queue 1034 by NVMe device 206 and / or native NVMe driver 1032. In the illustrated example, if the translated command has not yet been serviced, BE block service driver 1012 waits for the service of the translated command to complete. When the translated command has been serviced, control proceeds to box 1216, where example converter 1046 translates the completion status entry from I / O queue 1034 (box 1216). For example, queue interface 1044 accesses the completion status entry from I / O queue 1034's IOCQ, and converter 1046 translates the completion status entry by converting one or more physical parameters into one or more corresponding virtual parameters for use by guest VM 1002. In some examples, the completion status entry indicates a DMA operation that copies data from NVMe device 206 to guest memory buffer 1022 corresponding to guest VM 1002 / copys data from guest memory buffer 1022 corresponding to guest VM 1002 to NVMe device 206 (e.g., Figure 10 The DMA operation 1018 is completed. Example buffer interface 1042 submits the translated completion status entry to shared ring buffer 1026 (box 1218). Example notifyer 1048 ( Figure 10 The notifier 1048 notifies the guest VM 1002 of the completion (box 1220). For example, the notifier 1048 sends the command completion notification to the PVIO FE block driver 1008 of the guest VM 1002 via a shared ring buffer 1026 and / or asserts a virtual interrupt to the PVIO FE block driver 1008. Figure 10 The example process has ended.

[0079] Figure 10 It is capable of execution Figure 10 and Figures 6-9 The instructions are to implement the example ZCBV-MPT technology and / or be able to perform the instructions disclosed herein. Figure 11 The block diagram of an example processor platform 1300 is shown, illustrating instructions for implementing the example ZCBV-PVIO technology disclosed herein. The processor platform 1300 can be, for example, a server, a personal computer, a mobile device (e.g., a cellular phone, a smartphone, etc.). Tablet devices such as tablets, personal digital assistants (PDAs), internet devices, game consoles, set-top boxes, or any other type of computing device.

[0080] The processor platform 1300 of the illustrated example includes one or more processors 1312. The processor(s) 1312 of the illustrated example are hardware. For example, the processor(s) 1312 can be implemented by one or more integrated circuits, logic circuits, microprocessors or controllers from any desired family or manufacturer. The hardware processor(s) can be semiconductor based, such as silicon based. To implement the ZCBV-MPT techniques disclosed herein, the processor(s) 1012 of the illustrated example implement one or more of an example guest native driver 214a, 214b Figure 12 ), an example virtual NVMe device 224 ​ ), an example guest queue manager 216 ​ ), an example mediator 218 ​ ), an example shadow queue manager 220 ​ ), an example host native NVMe driver 222 ​ ), and / or an example ATT 228 ​ ). To implement the ZCBV-PVIO techniques disclosed herein, the processor(s) 1012 of the illustrated example implement one or more of an example PVIO FE block driver 1008 ​ ), an example BE block service driver 1012 ​ ), an example buffer interface 1042 ​ ), an example queue interface 1044 ​ ), an example translator 1046 ​ ), an example notifier 1048 ​ ), and / or an example native NVMe driver 1032 ​ ).

[0081] The processor(s) 1312 of the illustrated example include local memory 1313 (e.g., cache). The processor(s) 1312 of the illustrated example, via bus 1318, communicate with a main memory including volatile memory 1314 and non-volatile memory 1316. The volatile memory 1314 can be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS dynamic random access memory (RDRAM), and / or any other type of random access memory device. The non-volatile memory 1316 can be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1314, 1316 is controlled by a memory controller.

[0082] The processor platform 1300 of the illustrated example also includes an interface circuit 1320. The interface circuit 1320 can be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), and / or a PCI express interface.

[0083] In the illustrated example, one or more input devices 1322 are connected to the interface circuit 1320. The input device(s) 1322 permit(s) a user to enter data and / or commands into the processor 1312. The input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, isopoint and / or a voice recognition system.

[0084] One or more output devices 1324 are also connected to the interface circuit 1320 of the illustrated example. The output devices 1324 can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display, a cathode ray tube display (CRT), a touchscreen, a tactile output device, a printer and / or a speaker). The interface circuit 1320 of the illustrated example, thus, typically includes a graphics driver card, a graphics driver chip and / or a graphics driver processor.

[0085] The interface circuit 1320 of the illustrated example also includes a communication device such as a transmitter, a receiver, a transceiver, a modem and / or a network interface card to facilitate exchange of data with external machines (e.g., computing devices of any kind) via a network 1326 (e.g., an Ethernet connection, a digital subscriber line (DSL), a telephone line, coaxial cable, a cellular telephone system, etc.).

[0086] The processor platform 1300 of the illustrated example also includes one or more mass storage devices 1328 for storing software and / or data. Examples of such mass storage devices 1328 include floppy

[0087] Implementations ​ , ​ and / or ​ Exemplary machine readable instructions 1332 to implement such techniques can be stored on the mass storage device 1328, variously loaded into the volatile memory 1314, stored within the non-volatile memory 1316, and / or stored on a removable storage medium such as a CD or DVD.

[0088] From the foregoing, it will be appreciated that the example methods, apparatus, and articles of manufacture disclosed herein use techniques to improve virtualization performance associated with accessing virtualized storage and / or storage space from virtual machines. Existing I / O virtualization techniques include direct pass-thru techniques, single root input / output virtualization (SR-IOV), and paravirtualization. Existing direct pass-thru techniques cannot be used to share a single physical device across multiple guest VMs, and thus their use is limited to virtualization configurations in which an entire hardware resource is assigned exclusively to a single guest VM. SR-IOV can share a single physical device across several guest VMs. However, SR-IOV techniques require custom hardware extensions. Thus, SR-IOV techniques are limited to hardware-based implementations. Such hardware-based implementations can be significantly costly based, for example, on how many virtual functions are to be supported in the SR-IOV hardware. Due to the hardware implementations, scalability is poor since new hardware needs to be designed / manufactured when new virtual functions are added. Paravirtualization I / O techniques provide a hardware-neutral interface to guest VMs. However, paravirtualization requires the host machine's CPU to handle bulk data transfers between memory locations. As a result, during memory-intensive processes, paravirtualization can overload the host machine's CPU resources.

[0089] The example ZCBV techniques disclosed herein improve virtualization performance associated with accessing data in physical resources from guest VMs. For example, the ZCBV techniques disclosed herein eliminate the need to perform data copy operations on the VMM back-end side of a virtualization system. In this way, the host machine's CPU is not required to handle bulk data transfers. Instead, the examples disclosed herein employ DMA data transfers to copy data between memory locations in response to data access requests from guest VMs. As a result, the example ZCBV techniques disclosed herein improve the efficiency of block device I / O virtualization. In addition to reducing the use of CPU cycles (e.g., for performing copy operations of bulk data between NVMe memory and guest VM memory space), the example ZCBV techniques disclosed herein improve responsiveness (e.g., reduce latency) and / or increase data transfer speeds of virtual resources (e.g., virtual data storage resources based on underlying physical NVMe data storage resources). For example, using examples disclosed herein with 3D cross-point memory (e.g., implemented in Optane Optane TM memory), data transfer speeds equal to or greater than 2000 megabytes per second (MB / s) can be achieved. In other implementations, such as when used with other types of NV memory devices, examples disclosed herein can be used to achieve other data transfer speeds.

[0090] The following relates to other examples disclosed herein.

[0091] Example 1 is an apparatus for processing commands from a virtual machine. The apparatus of example 1 includes a guest queue manager to be located in a virtual non-volatile memory device of a virtual machine monitor executing on one or more processors, the guest queue manager to access a first command submitted to a guest queue by a native non-volatile memory driver executing in a guest virtual machine; a mediator to generate a translated command based on the first command by translating a virtual parameter of the first command to a physical parameter associated with a physical non-volatile memory device; a shadow queue manager to submit the translated command to a shadow queue to be processed by the physical non-volatile memory device based on the physical parameter; and the guest queue manager to submit a completion status entry to the guest queue, the completion status entry indicating completion of a direct memory access operation to copy data between the physical non-volatile memory device and a guest memory buffer corresponding to the guest virtual machine.

[0092] In example 2, the subject matter of example 1 can optionally include that the translated command is processed by the physical non-volatile memory device after synchronizing the translated command from the shadow queue in the virtual machine monitor to a physical queue in the physical non-volatile memory device.

[0093] In example 3, the subject matter of any one of examples 1-2 can optionally include that the first command is at least one of a management command or an input / output command, the management command to at least one of manage a queue, obtain driver configuration information, or set driver configuration information, and the input / output command to access data in a memory.

[0094] In example 4, the subject matter of any one of examples 1-3 can optionally include that the virtual parameter includes a virtual memory address of data and the physical parameter includes a physical memory address of the data.

[0095] In example 5, the subject matter of any one of examples 1-4 can optionally include that the virtual parameter includes a guest queue identifier of the guest queue and the physical parameter includes a host queue identifier of the shadow queue.

[0096] In example 6, the subject matter of any one of examples 1-5 can optionally include that the shadow queue manager is to create the shadow queue prior to submitting the translated command to the shadow queue.

[0097] In Example 7, the subject matter of any one of Examples 1-6 can optionally include that the guest queue manager is further to determine, prior to translating the first command, that the first command is to be processed by the physical non-volatile memory device, the determination based on the first command being an I / O command to request data from the physical non-volatile memory device or the first command being a management command to access a configuration of the physical non-volatile memory device.

[0098] In Example 8, the subject matter of any one of Examples 1-7 can optionally include that the virtual non-volatile memory device is a virtual non-volatile memory express (NVMe) device and the physical non-volatile memory device is a physical NVMe device.

[0099] In Example 9, the subject matter of any one of Examples 1-8 can optionally include a memory; one or more processors in circuit with the memory; and a network interface in circuit with the one or more processors, the one or more processors to execute the guest queue manager, the mediator, and the shadow queue manager.

[0100] Example 10 is a non-transitory computer-readable storage medium including instructions that when executed, cause one or more processors to at least: access, by a virtual non-volatile memory device in a virtual machine monitor, a first command submitted to a guest queue by a native non-volatile memory driver executing in a guest virtual machine; generate, by the virtual non-volatile memory device, a translated command based on the first command by translating virtual parameters of the first command to physical parameters associated with a physical non-volatile memory device; submit, by the virtual non-volatile memory device, the translated command to a shadow queue to be processed by the physical non-volatile memory device based on the physical parameters; and submit, by the virtual non-volatile memory device, a completion status entry to the guest queue, the completion status entry indicating completion of a direct memory access operation to copy data between the physical non-volatile memory device and a guest memory buffer corresponding to the guest virtual machine.

[0101] In Example 11, the subject matter of Example 10 can optionally include that the translated command is processed by the physical non-volatile memory device after synchronizing the translated command from the shadow queue in the virtual machine monitor to a physical queue in the physical non-volatile memory device.

[0102] In Example 12, the subject matter of any one of Examples 10-11 can optionally include that the first command is at least one of a management command to at least one of manage a queue, obtain driver configuration information, or set driver configuration information or an input / output command to access data in a memory.

[0103] In Example 13, the subject matter of any one of Examples 10-12 can optionally include that the virtual parameters comprise virtual memory addresses of the data, and the physical parameters comprise physical memory addresses of the data.

[0104] In Example 14, the subject matter of any one of Examples 10-13 can optionally include that the virtual parameters comprise guest queue identifiers of the guest queues, and the physical parameters comprise host queue identifiers of the shadow queues.

[0105] In Example 15, the subject matter of any one of Examples 10-14 can optionally include that the instructions further cause the one or more processors to create the shadow queue through a virtual non-volatile memory device prior to submitting the converted command to the shadow queue.

[0106] In Example 16, the subject matter of any one of Examples 10-15 can optionally include that the instructions further cause the one or more processors to determine, prior to converting the first command, that the first command is to be processed by a physical non-volatile memory device, the determination being based on the first command being an I / O command that requests data from the physical non-volatile memory device, or the first command being a management command to access a configuration of the physical non-volatile memory device.

[0107] In Example 17, the subject matter of any one of Examples 10-16 can optionally include that the virtual non-volatile memory device is a virtual non-volatile memory express (NVMe) device, and the physical non-volatile memory device is a physical NVMe device.

[0108] Example 18 is a method for processing commands from a virtual machine. The method of Example 18 includes accessing, by a virtual non-volatile memory device in a virtual machine monitor executing on one or more processors, a first command submitted to a guest queue by a native non-volatile memory driver executing in a guest virtual machine; generating, by the virtual non-volatile memory device, a converted command based on the first command by converting virtual parameters of the first command to physical parameters associated with a physical non-volatile memory device; submitting, by the virtual non-volatile memory device, the converted command to a shadow queue to be processed by the physical non-volatile memory device based on the physical parameters; and submitting, by the virtual non-volatile memory device, a completion status entry to the guest queue, the completion status entry indicating completion of a direct memory access operation to copy data between the physical non-volatile memory device and a guest memory buffer corresponding to the guest virtual machine.

[0109] In Example 19, the subject matter of Example 18 can optionally include that the converted command is processed by the physical non-volatile memory device after synchronizing the converted command from the shadow queue in the virtual machine monitor to a physical queue in the physical non-volatile memory device.

[0110] In Example 20, the subject matter of any one of Examples 18-19 can optionally include that the first command is at least one of a management command or an input / output command, the management command to at least one of manage a queue, obtain drive configuration information, or set drive configuration information, and the input / output command to access data in a memory.

[0111] In Example 21, the subject matter of any one of Examples 18-20 can optionally include that the virtual parameter includes a virtual memory address of the data, and the physical parameter includes a physical memory address of the data.

[0112] In Example 22, the subject matter of any one of Examples 18-21 can optionally include that the virtual parameter includes a guest queue identifier of a guest queue, and the physical parameter includes a host queue identifier of a shadow queue.

[0113] In Example 23, the subject matter of any one of Examples 18-22 can optionally include that the shadow queue is created by the virtual non-volatile memory device prior to submitting the converted command to the shadow queue.

[0114] In Example 24, the subject matter of any one of Examples 18-23 can optionally include that the determining that the first command is to be handled by the physical non-volatile memory device is based on the first command being an I / O command requesting data from the physical non-volatile memory device, or the first command being a management command to access a configuration of the physical non-volatile memory device.

[0115] In Example 25, the subject matter of any one of Examples 18-24 can optionally include that the virtual non-volatile memory device is a virtual non-volatile memory express (NVMe) device, and the physical non-volatile memory device is a physical NVMe device.

[0116] Example 26 is an apparatus for processing commands from a virtual machine. The apparatus of Example 26 includes a command accessing means for accessing a first command submitted to a guest queue by a native non-volatile memory driver executing in a guest virtual machine, the command accessing means located in a virtual non-volatile memory device of a virtual machine monitor executing on one or more processors; a translating means for generating a translated command based on the first command by translating a virtual parameter of the first command to a physical parameter associated with a physical non-volatile memory device; a command submitting means for submitting the translated command to a shadow queue to be processed by the physical non-volatile memory device based on the physical parameter; and a completion submitting means for submitting a completion status entry to the guest queue, the completion status entry indicating completion of a direct memory access operation to copy data between the physical non-volatile memory device and a guest memory buffer corresponding to the guest virtual machine.

[0117] In Example 27, the subject matter of claim 26 can optionally include that the translated command is processed by the physical non-volatile memory device after synchronizing the translated command from the shadow queue in the virtual machine monitor to a physical queue in the physical non-volatile memory device.

[0118] In Example 28, the subject matter of any one of claims 26-27 can optionally include that the first command is at least one of a management command or an input / output command, the management command for at least one of managing a queue, obtaining driver configuration information, or setting driver configuration information, and the input / output command for accessing data in a memory.

[0119] In Example 29, the subject matter of any one of claims 26-28 can optionally include that the virtual parameter includes a virtual memory address of data and the physical parameter includes a physical memory address of the data.

[0120] In Example 30, the subject matter of any one of claims 26-29 can optionally include that the virtual parameter includes a guest queue identifier of the guest queue and the physical parameter includes a host queue identifier of the shadow queue.

[0121] In Example 31, the subject matter of any one of claims 26-30 can optionally include a queue creating means for creating the shadow queue prior to submitting the translated command to the shadow queue.

[0122] In Example 32, the subject matter of any one of Examples 26-31 can optionally include the command intercepting means for determining, prior to translating the first command, that the first command is to be processed by the physical non-volatile memory device, determining based on the first command being an I / O command requesting data from the physical non-volatile memory device, or the first command being a management command for accessing a configuration of the physical non-volatile memory device.

[0123] In Example 33, the subject matter of any one of Examples 26-32 can optionally include a memory; one or more processors in circuit with the memory; and a network interface in circuit with the one or more processors, the one or more processors to execute the command accessing means, the translating means, the command submitting means, and the completion submitting means.

[0124] Example 34 is an apparatus for processing commands from a virtual machine. The apparatus of Example 34 includes a buffer interface in an input / output virtual machine executing on one or more processors to access a first command submitted to a buffer by a paravirtualized input / output front-end block driver executing in a guest virtual machine; a translator to generate a translated command based on the first command by translating a virtual parameter of the first command to a physical parameter associated with a physical resource; a queue interface to submit the translated command to an input / output queue to be processed by the physical resource based on the physical parameter; and the buffer interface to submit a completion status entry to the buffer indicating completion of a direct memory access operation to copy data between the physical resource and a guest memory buffer corresponding to the guest virtual machine.

[0125] In Example 35, the subject matter of Example 34 can optionally include the queue interface further to create the input / output queue prior to submitting the translated command to the input / output queue.

[0126] In Example 36, the subject matter of any one of Examples 34-35 can optionally include the physical resource is a non-volatile memory express device.

[0127] In Example 37, the subject matter of any one of Examples 34-36 can optionally include the virtual parameter includes at least one of a virtual memory address or a shared ring buffer identifier, and the physical parameter includes at least one of a physical memory address or a physical queue identifier.

[0128] In Example 38, the subject matter of any one of Examples 34-37 can optionally include the physical resource is a non-volatile memory express device.

[0129] In Example 39, the subject matter of any one of Examples 34-38 can optionally include a notifier to notify the guest virtual machine of completion of the first command.

[0130] In Example 40, the subject matter of any one of Examples 34-39 can optionally include: a memory; one or more processors in circuit with the memory; and a network interface in circuit with the one or more processors; the one or more processors to execute the buffer interface, the translator, and the queue interface.

[0131] Example 41 is a non-transitory computer-readable storage medium including instructions that when executed cause one or more processors to at least: access a first command submitted by a paravirtualized input / output front-end block driver executing in a guest virtual machine to a buffer; generate a translated command based on the first command by translating a virtual parameter of the first command to a physical parameter associated with a physical resource; submit the translated command to an input / output queue to be processed by the physical resource based on the physical parameter; and submit a completion status entry to the buffer indicating completion of a direct memory access operation to copy data between the physical resource and a guest memory buffer corresponding to the guest virtual machine.

[0132] In Example 42, the subject matter of Example 41 can optionally include the instructions further cause the one or more processors to create the input / output queue prior to submitting the translated command to the input / output queue.

[0133] In Example 43, the subject matter of any one of Examples 41-42 can optionally include the physical resource is a non-volatile memory express device.

[0134] In Example 44, the subject matter of any one of Examples 41-43 can optionally include the virtual parameter includes at least one of a virtual memory address or a shared ring buffer identifier, and the physical parameter includes at least one of a physical memory address or a physical queue identifier.

[0135] In Example 45, the subject matter of any one of Examples 41-44 can optionally include the physical resource is a non-volatile memory express device.

[0136] In Example 46, the subject matter of any one of Examples 41-45 can optionally include the instructions further cause the one or more processors to notify the guest virtual machine of completion of the first command.

[0137] Example 47 is a method for processing commands from a virtual machine. The method of Example 47 includes accessing, by a back-end block service driver in an input / output virtual machine executing on one or more processors, a first command submitted to a buffer by a front-end block driver executing in a guest virtual machine; generating, by the back-end block service driver based on the first command, a translated command by translating a virtual parameter of the first command to a physical parameter associated with a physical resource; submitting, by the back-end block service driver based on the physical parameter, the translated command to an input / output queue to be processed by the physical resource; and submitting, by the back-end block service driver, a completion status entry to the buffer indicating completion of a direct memory access operation to copy data between the physical resource and a guest memory buffer corresponding to the guest virtual machine.

[0138] In Example 48, the subject matter of claim 47 can optionally include creating, by at least one of the back-end block service driver or a native device driver, the input / output queue prior to submitting the translated command to the input / output queue.

[0139] In Example 49, the subject matter of any one of claims 47-48 can optionally include that the physical resource is a non-volatile memory express device.

[0140] In Example 50, the subject matter of any one of claims 47-49 can optionally include that the virtual parameter includes at least one of a virtual memory address or a shared ring buffer identifier, and the physical parameter includes at least one of a physical memory address or a physical queue identifier.

[0141] In Example 51, the subject matter of any one of claims 47-50 can optionally include that the physical resource is a non-volatile memory express device.

[0142] In Example 52, the subject matter of any one of claims 47-51 can optionally include notifying, by the back-end block service driver, the guest virtual machine of completion of the first command.

[0143] Example 53 is an apparatus for processing a command from a virtual machine. The apparatus of Example 53 includes a command accessing means, in an input / output virtual machine executing on one or more processors, to access a first command submitted to a buffer by a paravirtualized input / output front-end block driver executing in a guest virtual machine; a translating means to generate a translated command based on the first command by translating a virtual parameter of the first command to a physical parameter associated with a physical resource; a command submitting means to submit the translated command to an input / output queue to be processed by the physical resource based on the physical parameter; and a completion submitting means to submit a completion status entry to the buffer indicating completion of a direct memory access operation to copy data between the physical resource and a guest memory buffer corresponding to the guest virtual machine.

[0144] In Example 54, the subject matter of claim 53 can optionally include a queue creating means to create the input / output queue prior to submitting the translated command to the input / output queue.

[0145] In Example 55, the subject matter of any one of claims 53-54 can optionally include that the physical resource is a non-volatile memory express device.

[0146] In Example 56, the subject matter of any one of claims 53-55 can optionally include that the virtual parameter includes at least one of a virtual memory address or a shared ring buffer identifier, and the physical parameter includes at least one of a physical memory address or a physical queue identifier.

[0147] In Example 57, the subject matter of any one of claims 53-56 can optionally include that the physical resource is a non-volatile memory express device.

[0148] In Example 58, the subject matter of any one of claims 53-57 can optionally include a completion status notifying means to notify the guest virtual machine of completion of the first command.

[0149] In Example 59, the subject matter of any one of claims 53-58 can optionally include a memory; one or more processors in circuit with the memory; and a network interface in circuit with the one or more processors, the one or more processors to execute the command accessing means, the translating means, the command submitting means, and the completion submitting means.

[0150] Although certain example methods, apparatus and articles of manufacture have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus and articles of manufacture falling within the scope of the claims.

Claims

1. An apparatus for processing commands from a virtual machine, the apparatus comprising: a guest queue manager to reside in a virtual non-volatile memory device of a virtual machine monitor executing on one or more processors, the guest queue manager to access a first command submitted to a guest queue by a native non-volatile memory driver executing in a guest virtual machine; a mediator to generate a translated command based on the first command by translating a virtual parameter of the first command to a physical parameter associated with a physical non-volatile memory device; a shadow queue manager to submit the translated command to a shadow queue to be processed by the physical non-volatile memory device based on the physical parameter; and the guest queue manager to submit a completion status entry to the guest queue, the completion status entry to indicate completion of a direct memory access operation to copy data between the physical non-volatile memory device and a guest memory buffer corresponding to the guest virtual machine, wherein the shadow queue manager is to create the shadow queue prior to submitting the translated command to the shadow queue.

2. The apparatus of claim 1, wherein, the translated command is processed by the physical non-volatile memory device after synchronizing the translated command from the shadow queue in the virtual machine monitor to a physical queue in the physical non-volatile memory device.

3. The apparatus of claim 1, wherein, the first command is at least one of a management command to at least one of manage a queue, obtain driver configuration information, or set driver configuration information, or an input / output command to access data in a memory.

4. The apparatus of claim 1, wherein, the virtual parameter includes a virtual memory address of the data, and the physical parameter includes a physical memory address of the data.

5. The apparatus of claim 1, wherein, the virtual parameter includes a guest queue identifier of the guest queue, and the physical parameter includes a host queue identifier of the shadow queue.

6. The apparatus of claim 1, wherein, the guest queue manager is further to determine that the first command is to be processed by the physical non-volatile memory device prior to translating the first command, the determination based on the first command being an I / O command to request data from the physical non-volatile memory device, or the first command being a management command to access a configuration of the physical non-volatile memory device.

7. The apparatus of claim 1, wherein, the virtual non-volatile memory device is a virtual non-volatile memory express (NVMe) device, and the physical non-volatile memory device is a physical NVMe device.

8. The apparatus of claim 1, further comprising: a memory; one or more processors in circuit with the memory; and a network interface in circuit with the one or more processors, the one or more processors to execute the guest queue manager, the mediator, and the shadow queue manager.

9. A non-transitory computer-readable storage medium comprising instructions that, when executed, cause one or more processors to at least: access, by a virtual non-volatile memory device in a virtual machine monitor, a first command submitted to a guest queue by a native non-volatile memory driver executing in a guest virtual machine; generate, by the virtual non-volatile memory device, a translated command based on the first command by translating a virtual parameter of the first command to a physical parameter associated with a physical non-volatile memory device; submit, by the virtual non-volatile memory device, the translated command to a shadow queue to be processed by the physical non-volatile memory device based on the physical parameter; and submit, by the virtual non-volatile memory device, a completion status entry to the guest queue indicating completion of a direct memory access operation to copy data between the physical non-volatile memory device and a guest memory buffer corresponding to the guest virtual machine, wherein the instructions further cause the one or more processors to create, by the virtual non-volatile memory device, the shadow queue prior to submitting the translated command to the shadow queue.

10. The non-transitory computer-readable storage medium of claim 9, wherein, the translated command is processed by the physical non-volatile memory device after synchronizing the translated command from the shadow queue in the virtual machine monitor to a physical queue in the physical non-volatile memory device.

11. The non-transitory computer-readable storage medium of claim 9, wherein, the first command is at least one of a management command to at least one of manage a queue, obtain driver configuration information, or set driver configuration information, or an input / output command to access data in memory.

12. The non-transitory computer-readable storage medium of claim 9, wherein, the virtual parameter comprises a virtual memory address of the data, and the physical parameter comprises a physical memory address of the data.

13. The non-transitory computer-readable storage medium of claim 9, wherein, the virtual parameter comprises a guest queue identifier of the guest queue, and the physical parameter comprises a host queue identifier of the shadow queue.

14. The non-transitory computer-readable storage medium of claim 9, wherein, the instructions further cause the one or more processors to determine, prior to translating the first command, that the first command is to be processed by the physical non-volatile memory device based on the first command being an I / O command to request data from the physical non-volatile memory device or the first command being a management command to access a configuration of the physical non-volatile memory device.

15. The non-transitory computer-readable storage medium of claim 9, wherein, the virtual non-volatile memory device is a virtual non-volatile memory express (NVMe) device, and the physical non-volatile memory device is a physical NVMe device.

16. A method for processing commands from a virtual machine, the method comprising: accessing, by a virtual non-volatile memory device in a virtual machine monitor executing on one or more processors, a first command submitted to a guest queue by a native non-volatile memory driver executing in a guest virtual machine; generating, by the virtual non-volatile memory device, a translated command from the first command by translating a virtual parameter of the first command to a physical parameter associated with a physical non-volatile memory device based on the first command; submitting, by the virtual non-volatile memory device, the translated command to a shadow queue to be processed by the physical non-volatile memory device based on the physical parameter; and submitting, by the virtual non-volatile memory device, a completion status entry to the guest queue indicating completion of a direct memory access operation to copy data between the physical non-volatile memory device and a guest memory buffer corresponding to the guest virtual machine, wherein the method further comprises creating, by the virtual non-volatile memory device, the shadow queue prior to submitting the translated command to the shadow queue.

17. The method of claim 16, wherein, the translated command is processed by the physical non-volatile memory device after synchronizing the translated command from the shadow queue in the virtual machine monitor to a physical queue in the physical non-volatile memory device.

18. The method of claim 16, wherein, the first command is at least one of a management command to at least one of manage a queue, obtain drive configuration information, or set drive configuration information, or an input / output command to access data in a memory.

19. The method of claim 16, wherein, the virtual parameter comprises a virtual memory address of the data, and the physical parameter comprises a physical memory address of the data.

20. The method of claim 16, wherein, the virtual parameter comprises a guest queue identifier of the guest queue, and the physical parameter comprises a host queue identifier of the shadow queue.

21. The method of claim 16, further comprising determining, prior to translating the first command, that the first command is to be processed by the physical non-volatile memory device, the determining based on the first command being an I / O command to request data from the physical non-volatile memory device or the first command being a management command to access a configuration of the physical non-volatile memory device.

22. The method of claim 16, wherein, the virtual non-volatile memory device is a virtual non-volatile memory express (NVMe) device, and the physical non-volatile memory device is a physical NVMe device.

Images