Apparatus and method for providing quality of service on a virtual interface of a solid-state storage device

Abstract

Embodiments of the present disclosure relate to devices and methods for providing quality of service on a virtual interface of a solid-state storage device. The present disclosure describes several aspects of providing quality of service (QoS) through a virtual interface of solid-state storage. In some aspects, a storage media switch receives an input / output (I / O) command for data access from a host device, including an identifier of a virtual interface associated with a namespace of accessible solid-state storage. The switch determines a certain amount of data that the I / O command will access through the namespace. It is determined whether the amount of data that the I / O command will access through the namespace exceeds a predetermined threshold for accessing data through the namespace. When it is determined that the amount of data does not exceed the predetermined threshold, the I / O command is issued to the solid-state storage to make the data accessible. In this way, the switch can provide QoS for solid-state storage with virtual access based on the access parameters of the namespace.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to U.S. Provisional Patent Application Serial No. 62 / 715,706, filed on August 7, 2018, and U.S. Provisional Patent Application Serial No. 62 / 716,278, filed on August 8, 2018, the disclosures of which are incorporated herein by reference in their entireties. Technical Field

[0003] Embodiments of the present disclosure relate to apparatus and methods for providing quality of service over a virtual interface to a solid-state storage device. Background Art

[0004] Large-scale software development and cloud services are often implemented through single-tenant or multi-tenant architectures, which can provide scalability, cost, and security advantages through common systems running on shared resources. Typically in a multi-tenant architecture, a single instance of a software application and associated hardware infrastructure can serve multiple tenants or clients that share the software application, related data resources, and underlying hardware infrastructure. In this way, multiple tenants of a data center or cloud can simultaneously access various computing resources (such as processors and memory), network resources, and storage resources through a hypervisor that exposes these resources for use by the software tenants of the data center or cloud.

[0005] However, resource access through a hypervisor or other software layer may be complex and difficult to implement to some extent to ensure that the access provided meets the different service level requirements of each tenant. For example, some enterprise-level tenants may require high priority and high bandwidth access to storage resources, while other consumer-level tenants have very loose requirements for priority, bandwidth, or latency of storage resource access. With the development of aggregation and expansion of storage (such as non-dedicated storage devices), the hypervisor may have difficulty isolating or prioritizing tenants' corresponding access to aggregated storage resources. In such a case, active tenants may saturate the aggregated storage resources with multiple data access requests, which can delay or prevent other tenants from accessing the aggregated storage resources. Accordingly, the hypervisor or other host software often fails to provide corresponding access to aggregated storage resources in accordance with the different service level requirements of each tenant of a data center or cloud system. Summary of the invention

[0006] This Summary is provided to introduce subject matter that is further described in the Detailed Description and the Figures. Therefore, this Summary should not be considered to describe essential features nor be used to limit the scope of the claimed subject matter.

[0007] In some aspects, a storage media switch implements a method that receives an input / output (I / O) command for data access via a host interface and from a host device, the I / O command being associated with or linked to an identifier. The identifier of the I / O command is an identifier for a virtual interface or virtual function associated with a namespace, and the data of the solid-state storage device is accessible through the namespace. Based on the I / O command, the amount of data of the solid-state storage device that the I / O command will access through the namespace is determined. The method then determines whether the amount of data that the I / O command will access exceeds a predefined threshold (i.e., the number of bytes per period of time) for data access through the namespace within a period of time. In response to determining that the amount of data that the I / O command will access does not exceed the predefined threshold, the method issues the I / O command to the solid-state storage device via the storage media interface. By doing so, the method implemented by the storage media switch can provide quality of service (QoS) functionality to manage, measure or dispatch data access through the namespace in the solid-state storage device.

[0008] In other aspects, a device includes a hardware-based processor, a memory configured to maintain processor executable instructions to implement an application on the device, and a host interface configured to enable the application to access data in a solid-state storage device operably coupled to the device. The device also includes a storage medium switch that provides a storage medium interface for the solid-state storage device and a QoS manager that is implemented to receive I / O commands for data access via the host interface and from the application. The I / O command has an identifier of a virtual interface (i.e., a virtual function) that is associated with a namespace through which the data of the solid-state storage device is accessible. The QoS manager determines the amount of data of the solid-state storage device that the I / O command will access through the namespace based on the I / O command. The QoS manager then determines whether the amount of data that the I / O command will access through the namespace exceeds a predefined threshold for data access through the namespace within a period of time. In response to determining that the amount of data that the I / O command will access does not exceed the predefined threshold, the QoS manager issues the I / O command to the solid-state storage device via the storage medium interface of the switch.

[0009] In other aspects, a system on chip (SoC) is described as including a storage medium interface configured to enable access to a solid-state storage device and a host interface configured to enable communication with a host device. The SoC also includes a hardware-based processor and a memory configured to store processor executable instructions, which implements the QoS manager as an I / O command for data access via a host interface and received from a host. The I / O command includes an identifier of a virtual interface (i.e., a virtual function) associated with a namespace through which the data of the solid-state storage device is accessible. The QoS manager determines the amount of data of the solid-state storage device that the I / O command will access through the namespace based on the I / O command. The QoS manager then determines whether the amount of data that the I / O command will access exceeds a predefined threshold for data access through the namespace within a period of time (i.e., the operation of I / O per period of time). In response to determining that the amount of data that the I / O command will access does not exceed the predefined threshold, the QoS manager issues the I / O command to the solid-state storage device. Alternatively, in response to determining that the amount of data to be accessed by the I / O command reaches or exceeds a predefined threshold, the QoS manager will delay issuance of the I / O command to the solid-state storage device.

[0010] The details of one or more implementations are set forth in the accompanying drawings and the description that follows. Other features and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Details of one or more implementations of providing QoS on a virtual interface for a solid-state storage device are set forth in the accompanying drawings and the following detailed description. In the accompanying drawings, the leftmost digit of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different instances in the specification and the drawings indicates the same element:

[0012] Figure 1 An example operating environment having a device in which a virtual interface to a solid-state storage device is implemented is illustrated in accordance with one or more aspects of the present disclosure.

[0013] Figure 2 Picture shows Figure 1 An example configuration of a storage media switch and solid-state drives is shown in FIG.

[0014] Figure 3 An example configuration of a storage media switch associated with a host and a plurality of solid-state drives according to one or more aspects is illustrated.

[0015] Figure 4 Illustrated are example configurations of various storage media switch queues according to one or more aspects.

[0016] Figure 5 Illustrated are example configurations of QoS parameters that facilitate implementing QoS in accordance with one or more aspects.

[0017] Figure 6 An example method for automatically mapping host commands to virtual functions on a storage medium is depicted.

[0018] Fig. 7A An example method for selecting a virtual function of a storage medium is depicted.

[0019] Figure 7B Depicted are example methods for responding to host commands in accordance with one or more aspects of the present disclosure.

[0020] Figure 8 is a conceptual diagram illustrating the manipulation of an address field of a host command when the host command is mapped to a virtual function on a storage medium.

[0021] Fig. 9 An example method for providing QoS for solid-state storage devices accessed through a namespace is depicted.

[0022] Fig.10 An example method for submitting I / O commands to a solid-state storage device based on a bandwidth quota for a namespace is depicted.

[0023] Fig.11 An example method for managing data access of a virtual machine through a namespace of a solid-state storage device is described.

[0024] Fig.12 An example system-on-chip (SoC) environment is illustrated for implementing various aspects of QoS on a virtual interface to solid-state storage media.

[0025] Fig.13 An example storage media switch controller configured to implement various aspects of QoS on a virtual interface to solid-state storage media is illustrated. DETAILED DESCRIPTION

[0026] Conventional techniques for providing access to storage resources often rely on interface standards that are designed for legacy storage device types (i.e., rotating or optical media), storage area networks, or separate storage drives. However, as solid-state storage architectures develop, these conventional techniques are often limited or inefficient when extended to aggregated or virtualized storage volumes of solid-state media. For example, previous generations of servers provide storage access through a hypervisor or other software layer connected to a server client software interface to manage various parameters of each storage request leaving the host. Such software layers often cannot communicate directly with a solid-state storage media controller on the downstream or device side, through which the solid-state storage device can be accessed or addressed. Additionally, these legacy software layers may have further difficulties in virtualization of isolation or priority sorting of enabling corresponding tenants to access solid-state storage devices. Accordingly, hypervisors or other host software layers often cannot provide separate access to storage resources according to the different service level requirements of each tenant in a data center or cloud.

[0027] The present disclosure describes apparatus and techniques for providing QoS on a virtual interface for a solid-state storage device. Compared to conventional access techniques in which host software attempts to manage storage access, the described apparatus and techniques can implement a storage media switch to efficiently allocate bandwidth for a virtual solid-state storage device or access to a virtual solid-state storage device. Typically, the allocation or distribution of aggregate bandwidth in a storage device managed by a storage media switch can be provided as the quality of service (QoS) or management performance of storage access provided by the storage media switch. In other words, the storage media switch can control the amount of data consumed by a host, a virtual machine of the host, or a namespace of a storage device coupled to the storage media switch based on predefined parameters. Therefore, the aspects described herein enable administrators of a host or system to ensure that storage access is provided to clients and tenants to meet their required or desired service levels.

[0028] In various aspects, QoS is provided by a storage media switch through a single root I / O virtualization (SR-IOV) implementation of a peripheral component interconnect express (PCIe) interface of the switch, such as non-volatile memory express (NVMe). At a basic level, a SoC on SR-IOV can be based on distributing the available total bandwidth of the storage media switch across its backend solid-state storage devices (such as NVMe solid-state drives (SSDs)) to host virtual machines (VMs) mapped to virtual functions of the storage media switch (such as 34 PCIe physical / virtual functions exposed through SR-IOV).

[0029] In an example implementation, a controller of a storage media switch presents a physical SSD coupled to a storage interface as a single virtual disk (VD) to a host coupled to a host interface of the switch. The controller can segment the virtual disk into multiple configurable namespaces, such as namespaces of equal or different horizontal stripes. Each NVMe submission and completion queue associated with a VM of the host is provided access to only one of these namespaces or virtual disk segments. Based on this, each VM of the host can be assigned a portion of the total bandwidth based on a subscription policy (such as a data access parameter) that is adjusted by a QoS manager of the storage media switch.

[0030] By way of example, consider a QoS manager that is implemented as part of a command classifier of a storage media switch. The inbound engine of the storage media switch submits the fetched elements to the command classifier. The command classifier, or QoS manager, continuously tracks the bandwidth used by each namespace over a predefined period of time (i.e., a period of time or time interval). Upon receiving a command, the command classifier begins a command processing operation and determines whether the bandwidth quota of the destination namespace has been exceeded, or whether the object command will cause the bandwidth quota to be exceeded. If the quota for the namespace has been exceeded, the command and any additional commands for the namespace will be retained in the hierarchical queue, otherwise the command and other commands will be submitted to the submission queue of the solid-state device corresponding to the namespace. This is just one of many examples of QoS that the storage media switch described throughout the present disclosure can provide according to one or more aspects.

[0031] In various aspects of providing QoS on a virtual interface for a solid-state storage device, a storage media switch receives an input / output (I / O) command for data access from a host device. The I / O command for data access is associated with an identifier for a virtual interface, which is associated with a namespace through which the solid-state storage device can be accessed. The storage media switch determines the amount of data that the I / O command will access through the namespace. It is then determined whether the amount of data accessed by the I / O command through the namespace exceeds a predefined threshold for data access through the namespace. In response to determining that the amount of data does not exceed the predefined threshold, the I / O command is issued to the solid-state storage device to enable data access. Alternatively, in response to determining that the amount of data reaches or exceeds a predefined threshold, the storage media switch will delay the issuance of the I / O command, such as until another timing window begins, the inbound queue reaches an empty state, or the queue for the solid-state storage device reaches an empty state. By doing so, the switch can provide QoS for a virtually accessed solid-state storage device based on the access parameters of the namespace of the solid-state storage device.

[0032] The following discussion describes an operating environment, technologies that can be employed in the operating environment, and a system on a chip (SoC) in which components of the operating environment can be embodied.In the context of the present disclosure, reference is made to an operating environment by way of example only.

[0033] Operating Environment

[0034] Figure 1 The figure shows an example operating environment 100 with a host device 102 (referred to as a single "host 102"), where virtual function support or QoS on a virtual interface can be implemented according to one or more aspects. The host device 102 of the operating environment 100 can store or access various forms of data, files, objects, or information. Examples of host devices 102 may include computing clusters 104 (e.g., computing clusters of cloud 106), servers 108 or server hardware of data centers 110, or servers 112 (e.g., standalone), any of which may be configured as a storage network, storage service, or part of a cloud system. Other examples of host devices 102 (not shown) may include tablet computers, set-top boxes, data storage devices, wearable smart devices, televisions, content streaming devices, high-definition multimedia interface (HDMI) media sticks, smart appliances, home automation controllers, smart thermostats, Internet of Things (IoT) devices, mobile Internet devices (MIDs), network attached storage (NAS) drives, aggregate storage systems, server blades, game consoles, automotive entertainment devices, automotive computing systems, automotive control modules (e.g., engine or power control modules), etc. Generally, host device 102 may communicate or store data for any suitable purpose, such as to enable functionality of a particular type of device, provide a user interface, enable network access, implement gaming applications, play back media, provide navigation, edit content, provide data storage, etc.

[0035] The host device 102 includes a processor 114 and a computer-readable storage medium 116. The processor 114 can be implemented as any suitable type or number of processors (e.g., x86 or ARM), which can be single-core or multi-core, for executing instructions or commands of an operating system or other programs of the host device 102. The computer-readable medium 116 (CRM 116) includes a system memory 118, from which the virtual machine 120 of the host can be executed or implemented. The system memory 118 of the host device 102 can include any suitable type or combination of volatile memory or non-volatile memory. For example, the volatile memory of the host device 102 can include various types of random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), etc. The non-volatile memory can include a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), or a flash memory (e.g., NOR flash or NAND flash). These memories, alone or in combination, can store data associated with the applications, tenants, workloads, launchers, virtual machines, and / or operating systems of the host device 102.

[0036] In this example, the host device 102 includes a storage media switch 122 and a storage medium 124, which can be accessed through the storage media switch 122. Although shown as being combined with the host device 102, the storage media switch 122 and / or the storage medium 124 can be implemented separately from the host device 102 or remotely. The storage medium 124 of the host device 102 can be configured as any suitable type of data storage medium, such as a storage device, a storage drive, a storage array, a storage volume, etc. Although described with reference to the host device 102, the storage medium 124 can also be implemented separately as an independent device, or as part of a larger storage collection, such as a data center, a server cluster, or a virtualized storage system (e.g., for cloud-based storage or services). Examples of the storage medium 124 include a hard disk drive (HDD, not shown), an optical disk drive (not shown), a solid state drive 126 (SSD 126), or an array 126-0 to 126-m of m+1 SSDs.

[0037] Each SSD in SSD 126 includes or is formed by a non-volatile memory device, on which data or information of host device 102 or other sources is stored. Non-volatile memory devices can be implemented with solid-state memory media of any type or combination, such as flash memory, NAND flash memory, NAND memory, RAM, DRAM (e.g., for cache), SRAM, etc. In some cases, the data stored in the non-volatile memory device can be organized into files of data (e.g., content) or data objects stored in SSD 126 and accessed by host device 102 or tenants, workloads, or initiators of the host device. The type, size, or format of the file can vary according to the corresponding source, purpose, or application associated with the file. For example, the files stored in SSD 126 can include audio files, video files, text files, image files, multimedia files, electronic forms, etc.

[0038] In this example, the storage media switch 122 (switch 122) of the host device 102 is capable of supporting virtualization, providing QoS on a virtual interface, and enabling virtual functions associated with the storage media 124. In some aspects, the storage media switch 122 includes a virtual function (VF) address engine 128, a VF map 130, a quality of service (QoS) manager 132, and QoS parameters 134, each of which may be implemented to perform respective operations or functions for supporting virtualization, providing QoS, and enabling virtual functions associated with the storage media 124. The implementation and use of these entities vary and are described throughout this disclosure.

[0039] Typically, the VF address engine 128 can enable virtual function support for storage media that do not have inherent SR-IOV capabilities and for storage media that already support virtual functions. The VF address engine 128 processes host commands to automatically map host commands to appropriate virtual functions. In some examples, the VF address engine 128 can extract a virtual function identifier from a host command. In other examples where the storage medium 124 inherently supports SR-IOV, the VF address engine 128 can determine the virtual function identifier by looking up a routing identifier associated with the storage medium 124 from within the VF map 130. Using the VF map 130, the VF address engine 128 can determine the virtual function associated with the virtual function identifier. Using the virtual function identifier, the VF address engine 128 can select a virtual function of the storage medium 124 for executing the host command. In response to instructing the storage medium 124 to execute a transaction for implementing the host command, the VF address engine 128 can respond to the host command via the host interface.

[0040] In various aspects, QoS manager 132 may ensure that execution of host commands is managed to a quality of service level associated with a particular application, client, host, virtual machine of a host, tenant of a host, etc. For example, QoS manager 132 may determine a QoS for a virtual machine executing on host device 102 and ensure that I / O commands and data access transactions between host device 102 and storage medium 124 meet the determined QoS level. Alternatively or additionally, QoS manager 132 may measure and maintain values ​​of QoS parameters 134 that QoS manager 132 uses when responding to host commands.

[0041] Host device 102 may also include I / O ports 136, a graphics processing unit 138 (GPU), and a data interface 140. In general, I / O ports 136 allow host device 102 to interact with other devices, peripherals, or users. For example, I / O ports 136 may include or be coupled to a universal serial bus, a human interface device, audio input, audio output, etc. GPU 138 processes and renders graphics-related data for host device 102, such as user interface elements of an operating system, applications, etc. In some cases, GPU 138 accesses a portion of local memory to render graphics, or includes dedicated memory for rendering graphics for host device 102 (e.g., video RAM).

[0042] The data interface 140 of the host device 102 provides connectivity with one or more networks and other devices connected to those networks. The data interface 140 may include a wired interface, such as an Ethernet or fiber optic interface for communicating on a local network, an intranet, or the Internet. Alternatively or additionally, the data interface 140 may include a wireless interface that facilitates communication on a wireless network such as a wireless LAN, a wide area wireless network (e.g., a cellular network), and / or a wireless personal area network (WPAN). According to one or more aspects of the present disclosure, any of the data communicated through the I / O port 136 or the data interface 140 may be written to the storage medium 124 of the host device 102 or read from the storage medium 124 of the host device 102.

[0043] The data interface 140 can support the host interface of the storage media switch 122 and the storage media interface of the storage media switch 122. For example, the storage media switch 122 can receive host commands and respond to host commands from the host interface. The storage media switch 122 can indicate a transaction on the storage media interface to cause the storage medium 124 to execute the host command.

[0044] Figure 2 Picture shows Figure 11. An example configuration of a storage media switch 122 and SSD 126 shown in . In this example, the storage media switch 122 is operably coupled between a host 202 and SSDs 126-0 to 126-n (collectively referred to as "SSD 126"), from which storage virtualization areas, storage partitions, or storage segments (e.g., isolated storage areas) are provided. The storage media switch can be coupled to the host 202 and / or the SSDs via one or more corresponding PCIe interfaces (not shown), which can communicate in accordance with the NVMe protocol (e.g., NVMe rev 1.3). In this example, the host 202 (e.g., the host device 102) includes a plurality of virtual machines 120 executing on a computing resource 204 of the host. In general, the computing resource 204 of the host 202 may include a combination of processing resources and system memory of the host 202, which is used to implement applications, virtual machines, tenants, or initiators that access storage devices associated with the host 202.

[0045] According to one or more aspects, the storage media switch 122 can enable virtual functions on the storage media and / or provide QoS on the virtual functions for solid-state storage. In this example, the storage media switch 122 or its controller (not shown) enables access to the storage media associated with the switch through VFs 206-0 to 206-w (collectively referred to as "virtual functions 206"). In general, the virtual functions 206 are primarily concerned with data movement between the virtual machines (VMs) 120 of the host 202 and the storage provided by the SSD 126, and each virtual function 206 is associated with a physical function 208. Although one example of the physical function 208 is shown, the storage media switch 122 can be implemented with any number and / or combination of physical and virtual functions. Figure 2 The physical function 208 can be equivalent to one or more functions of a typical PCIe physical function device. In some cases, the physical function 208 is responsible for arbitration related to policy decisions, such as the link speed or network address used by the VM 120 in the case of networking, and for handling various input and output transactions between the storage media switch 122 and the VM 120 and SSD 126.

[0046] Typically, a tenant or initiator of a VM 120 or host 202 accesses data stored in SSD 126. In some implementations, storage media switch 122 presents aggregated storage media (such as SSD 126) to host 202 or the host's VM 120 as a virtual disk or storage volume. In the context of NVMe, the virtual disk may be segmented or partitioned into different areas accessed through namespaces. In other words, data access to a virtual disk or SSD 126 may be consumed through one or more namespaces corresponding to segments or stripes of the virtual disk or storage media. Figure 2 As shown, each of the SSDs 126 may be implemented with an SSD controller 210-0 to 210-n, and NAND channels 212-1 to 212-4 may be accessible through the SSD controllers 210-0 to 210-n. Each channel 212 of NAND (e.g., channel A or NAND channel 212) includes a plurality of NAND devices, which may be implemented as separate NAND devices or NAND dies of the SSD 126, which are accessible or addressable through corresponding NAND channels.

[0047] In some aspects, the storage media switch 122 manages access to the virtual function 206. For example, the storage media switch 122 can exchange information including host commands with the host 202, such as to facilitate data access between the SSDS 126 and the host memory. In some cases, the storage media switch 122 determines a virtual function identifier associated with the host command and automatically selects the virtual function 206 based on the virtual function identifier. Alternatively, the switch selects the virtual function 206 based on the virtual function 206 specified in the host command, for example, by matching the virtual function 206 with a driver request identifier of the SSDS 126.

[0048] The storage media switch 122 or a processor (not shown) of the switch 122 may also execute host commands of the virtual functions 206 and, after execution, respond to each of the host commands. By automatically mapping the virtual functions 206 in this manner, the storage media switch 122 enables SR-IOV for or on any of the SSDs 126, including any of the SSDs 126 that lack native support for virtual functions or SR-IOV-based features.

[0049] Figure 3An example configuration of a storage media switch 122 (switch 122) associated with a host and a plurality of solid-state drives, generally at 300, is illustrated, which is implemented according to one or more aspects of providing QoS on a virtual interface for a solid-state storage device. In this example, the switch 122 is operably coupled between a host 202 (not shown) and an array of SSDs 126, each of which is segmented into eight horizontal stripe namespaces (NS0 to NS31). Each of the SSDs 126 can be connected to the switch 122 via an NVMe interface, such as an NVMe interface implemented on a x4, x8, or x16 PCIe interface. Alternatively or additionally, the host 202 can be connected to the switch by an NVMe interface to enable data transactions through the switch 122 to the SSDs 126 or other storage devices.

[0050] Typically, in this example, the NVMe specification or protocol supports namespace-based access to storage media or SSDs. In various aspects of providing QoS, the virtual functions of the switch 122 are mapped to namespaces for segments or partitions of virtual disks. By doing so, QoS can be enabled and managed using namespaces so that VMs or tenants can subscribe to service levels for storage access, which are managed through namespace-based storage media access. In other words, a client or customer subscribes to a predefined amount of bandwidth and is connected to a host or through a virtual function of the host. The virtual function mapped to the namespace within the switch 122 is allocated bandwidth or access to the storage medium based on a threshold or quota assigned to the namespace, such as according to a subscription for data access at a specific service level. Within the NVMe subsystem of the switch 122, multiple isolation domains can be supported by the use of virtual functions, which are provided through SR-IOV on PCIe. By doing so, an administrator on the host can insert tenants into different address fields, such as VMs, and provide isolated and different access to storage devices, and the access and consumption of the storage devices are performed in the form of namespaces.

[0051] Back to Figure 3, the host 202 can be implemented as a multi-tenant host with any suitable number of virtual machines or tenants that can access corresponding NVMe command queues. In this example, the host 202 includes 32 VMs (not shown), each VM having a set of NVMe queues 302-0 to 302-31 of the switch 122, which are mapped to corresponding virtual functions 304 (VF0 to VF31, grouped as VF 304-0 to VF 304-31). These NVMe queues 302-0 to 302-31 can be implemented in the host memory and configured by the host as any suitable number of management queues (management queues) or I / O command queues. Here, each VM is assigned four queues, one of which is configured as a management queue and three are configured as I / O command queues.

[0052] In some aspects, the inbound and outbound queue engines of the hardware layer of the switch 122 can support up to 128 NVMe host submission queues (SQs) and host completion (CQ) queue pairs. Each of these queues can be individually mapped to any PCIe physical function or virtual function provided by the storage media switch 122. The host device or server implementing the VM can associate each host VM with a corresponding VF 304 within the switch 122. The VM can map the NVMe queue associated with the VM to the corresponding VF 304 within the switch 122 and / or further bind the NVMe queue to any SQ or CQ queue in the SQ or CQ queue provided by the switch 122. In other words, the VM manager (e.g., a server) running on the host can associate each host VM with a corresponding VF. Thereafter, the VM itself can map the NVMe queue in its space to the VF, thereby effectively causing the NVMe queue to be bound to any SQ or CQ in the SQ or CQ provided by the switch 122.

[0053] In terms of priority, submission queues within a VM can be assigned equal, higher, or lower priorities through an NVMe-based arbitration policy (fixed or weighted round-robin) that is available or configurable on a per-queue basis. This can be useful when one or more VMs use different submission queues from multiple submission queues to prioritize certain commands associated with a particular application or workload. Errors and exceptions for VF operations in this configuration are also isolated and detected on a per-VF basis. Therefore, errors originating from one VF of the switch 122 do not interfere with the I / O command flow of other VFs of the switch. Alternatively or additionally, selective reset of the state of a particular VF is enabled, such as a functional level reset, etc.

[0054] Referring to the command flow, in this example implementation for the switch 122, each VF 304 maps four universal command delivery (UCD) inbound queues. Alternatively, the architecture of the switch 122 has the flexibility to allocate a greater or lesser number of queues to each VF, as long as the total queue distribution of the VF does not exceed the total number of queues supported by the switch 122 (e.g., 128 queues). Typically, the host VM will submit a management command or an I / O command to the corresponding submission queue (SQ) and cause the corresponding UCD doorbell for the queue to ring. In response to the doorbell, the universal command delivery block (UCD block 306) can retrieve the SQ unit based on the preconfigured arbitration scheme of the switch 122. For example, if all I / O SQs are set up as equal priorities, the UCD block 306 will serve these inbound queues in a round-robin manner. From the UCD block 306 or the UCD inbound queue, the switch 122 submits the retrieved elements (e.g., management commands or I / O commands) to the command classifier processor (CCP or command processor 308). The retrieved elements may be submitted using a destination free list (DFL) of an I / O queue block 310 of a command processing subsystem 312 (subsystem 312) that includes or is associated with a command processor 308. Typically, the UCD block 306 will collate and distribute I / O commands 314 and management commands 316 to corresponding queues or blocks of the subsystem 312, which in this example also includes a management queue block 318 and a physical region page (PRP) list work queue block 320.

[0055] By way of example, consider Figure 4 , where an example configuration of the command processing subsystem 312 is generally shown at 400. Here, the UCD block 306 may submit commands to the command processor 308 using the destination free list (FDL) assigned to the queue being retrieved. Based on the arbitration priority assigned to the queue, the DFL generates an inbound completion queue element (IB_CQ) to the command processor 308. Typically, the command processor 308 or the QoS manager 132 monitors the inbound completion queue for new elements and initiates command processing operations upon receiving one.

[0056] In some aspects, the PRP list work queue enables access to the VF 304 through the switch 122. For example, the VF may use a PRP data-pointer including a virtual function (VF) identifier in the upper address for proper routing within the switch 122, the VF identifier being inserted by the processor or VF address engine. With respect to command routing, I / O commands that do not include or rely on physical region page (PRP) list retrieval (e.g., I / O commands less than 8Kb in size) will be resolved by the command processor 308 so that the PRP data pointer embedded in the I / O command can be processed before submission to the SSD device queue. Alternatively, through the PRP list work queue, I / O commands that include or rely on PRP list retrieval (e.g., I / O commands less than 8Kb in size) are routed to the management and execution processor 322 (MEP or management processor 322) so that the PRP list can be processed before submission to the SSD device queue.

[0057] In some aspects of providing QoS, the firmware of the command processor 308 or the QoS manager 132 tracks the bandwidth used by each namespace over a predefined duration. In some cases, the QoS manager 132 adds a logical block address (LBA) count per I / O per namespace as commands are processed by the command processor 308. Once the LBA count exceeds a predefined threshold or quota for a particular namespace, any additional commands for that namespace will be delayed or blocked in the corresponding staging queue (NSm_STAGE_Q) for each namespace.

[0058] After being delayed for at least a period of time, the commands stored in the hierarchical queue will be re-evaluated for submission to the SSD device submission queue (DEVn_SQ). In some cases, the commands are re-evaluated once a new timing window is started for servicing the namespace queue or retrieving the inbound command element. Alternatively, the re-evaluation can be initiated in response to the inbound completion queue (IBn_CQ) to the command processor or the device submission queue (DEVn_SQ) reaching an empty state. In some cases, the former is an indication that there are no new elements retrieved by the UCD block 306 that need to be evaluated during the current timing window.

[0059] To issue or submit an I / O command or element to a submission queue of a storage device, the command processor 308 should evaluate the various work queues in the following order: the namespace staging queue (NSm_STAGE_Q) corresponding to the device submission queue (DEVn_SQ), followed by the completion queue (MEP_CQ) of the management processor 322, and finally the inbound completion queue (IBn_CQ) or DFL queue of the I / O queue block. This order of queue priority can be reconfigured by the command processor 308 as needed, or to support alternative implementations of providing QoS through the switch 122.

[0060] Figure 5 An example configuration of QoS parameters 134 useful for implementing QoS according to one or more aspects is illustrated at 500. In this example, the QoS parameters 134 are organized with reference to namespaces 502-0 through 502-31, which may correspond to the QoS parameters used in Figure 3 506, and a host submission queue consumption index 508. Alternatively or additionally, the QoS parameters may include or reference a physical region page (PRP) list of an I / O command (e.g., for multiple LBAs or IOPs), a scatter-gather list of I / O commands (e.g., for multiple LBAs or IOPs), a plurality of I / O operations associated with an I / O command, or a logical block address (LBA) count of an I / O command.

[0061] Techniques for Providing QoS on Virtual Interfaces

[0062] The following discussion describes techniques for providing QoS on a virtual interface to a solid-state storage device, which can provide storage isolation, bandwidth control, and partitioning functionality to a host, a tenant, or a VM executing on a host. These techniques can be implemented using any of the environments and entities described herein, such as the VF address engine 128, the VF map 130, the QoS manager 132, or the QoS parameters 134. These techniques are included in Figure 6 , Fig. 7A and Figure 7B , and / or Figure 9-11 In the methods illustrated in FIG, each diagram is shown as a set of operations performed by one or more entities.

[0063] The methods are not necessarily limited to the order of operations shown in the associated figures. Rather, any of the operations may be repeated, skipped, replaced, or reordered to implement the various aspects described herein. Further, the methods may be used in conjunction with one another, in whole or in part, whether performed by the same entity, separate entities, or a combination thereof. For example, the methods may be combined to expose virtualized isolated areas of storage media while explicitly providing loss levels, load balancing, or data migration without host interaction or involvement. In the following discussion, by way of example, reference will be made to the following. Figure 1 The operating environment is 100 and Figure 2 , Figure 3 , Figure 4 and / or Figure 5 This reference is not intended to limit the described aspects to the operating environment 100, entity, or configuration, but rather to be an illustration of one example of various examples. Alternatively or additionally, the operations of these methods may also be performed by or with reference to Fig.12 System on Chip and / or Fig.13 This is achieved by using a storage media switch controller.

[0064] Figure 6 An example method 600 for automatically mapping host commands to virtual functions on a storage medium is depicted. The operations of the method 600 may be implemented by or with a storage medium switch 122 that includes a VF address engine 128 and uses a VF map 130.

[0065] At 602, the storage media switch 122 receives a host command from a host via a host interface of the storage media switch 122. For example, the VF address engine 128 of the storage media switch 122 may receive an indication of a new host command obtained from a host software application executing at virtual machine VM-0.

[0066] At 604, in response to receiving the host command, the storage media switch 122 determines a virtual function identifier associated with the host command. For example, an unused portion of a virtual address field of the host command may be populated with a sign-extended canonical portion of the address contained within the virtual address field. Relative to using the entire unused portion of the address field as the sign-extended canonical portion of the address, the storage media switch 122 may infer the virtual function identifier from the unused portion of the virtual address field and use the virtual function identifier to route the host command that will satisfy the appropriate virtual function.

[0067] As an example, a host command may include a 64-bit virtual address field, e.g., bits [63:0], even though only a portion of the virtual address field (e.g., bits [48:0]) may be used as a virtual address for the host command. Since some virtual address fields may not be used by the storage media switch 122 to address the host command, the unused portion of the virtual address field, e.g., bits [63:49], may include other information associated with the host command, such as a virtual function identifier, which the storage media switch 122 may use to execute the host command.

[0068] Some of the unused portions of the virtual address field may include a virtual function identifier. Storage media switch 122 may extract a virtual function identifier associated with the host command from a first unused portion of the virtual address field included in the host command. For example, bits [62:58] may include a virtual function identifier that storage media switch 122 uses to identify a particular virtual function associated with the command.

[0069] At 606, the storage media switch 122 selects a virtual function of the storage media 124 based on the virtual function identifier for executing the host command. For example, the VF address engine 128 may further isolate the tag portion of the address field of the host command to determine the host identifier. That is, some of the unused portions of the virtual address field may include the host identifier, and the storage media switch 122 may extract the host identifier associated with the host command from a second unused portion of the virtual address field included in the host command. For example, bit

[63] may include the host identifier, and the storage media switch 122 uses the host identifier to identify the specific host associated with the command.

[0070] The VF map 130 maintains an association or mapping between a virtual function identifier and a corresponding virtual function and a corresponding host. For example, the VF map 130 may maintain a mapping between a host identifier and a virtual function identifier to a host and a virtual function. Using the VF map 130, the VF address engine 128 uses the virtual function identifier and the host identifier to look up the corresponding virtual function in the case where the virtual function identifier and the host identifier have been determined from an unused portion of a virtual address contained in a host command, or from a canonical portion of a sign extension of a virtual address contained in a host command.

[0071] The VF address engine 128 may look up the specific host identifier (e.g., bit

[63] ) from the VF map 130 to determine which host is associated with the specific host identifier and, therefore, which host is the initiator of the host command. The VF address engine 128 may search the VF map 130 to determine a virtual function for the host command that matches the specific virtual function identifier (e.g., bits [62:58]).

[0072] At 608, the storage media switch 122 executes the host command using the virtual function assigned to the storage media 124. For example, using the host identifier and the virtual function identifier, the VF address engine 128 generates an internal interface selection value to route the host command to the intended virtual function at the storage media interface of the storage media switch 1212. The storage media switch 122 can use the interface selection value to ultimately route and cause the storage media 124 to perform a read / write transaction to satisfy the host command.

[0073] At 610, in response to performing a transaction to satisfy the host command, the storage media switch 122 responds to the host command via the host interface using the virtual function assigned to the storage medium 124. For example, in response to instructing the storage medium 124 to perform a transaction to satisfy the host command, the storage media switch 122 prepares a response to the host command by determining an original address for the host command. The VF address engine 128 can use only a portion of the internal routing address (e.g., without the interface selection value) to determine the original address for the host command. The VF address engine 128 can remove the bits containing the interface selection value from the routing address and use the remaining bits as the original address. For example, the original address may include bits [63:0], where bits [48:0] correspond to bits [48:0] of the original virtual address, and bits [63:49] correspond to a predefined value preferred by the host or a canonical sign-extended value of the original virtual address.

[0074] In the absence of any virtual function tag, the storage media switch 122 uses the original address to cause the host command to exit the host interface. In other words, the above operations show that the storage media switch 122 can determine the original address associated with the host command in response to the host. The storage media switch 122 determines the original address by removing the virtual function identifier tagged to the virtual address included in the host command, replacing the virtual function identifier with a predefined value preferred by the host, or a canonical portion of the sign extension of the remaining portion of the virtual address.

[0075] Fig. 7AAn example method 700A is depicted for selecting a virtual function of a storage medium for executing a host command based on a virtual function identifier that is encoded within the host command. The method 700A is an example of operations performed by the storage media switch 122 when step 606 is performed from the method 600 for automatically mapping a host command to a virtual function of a storage medium. The operations of the method 700A may be performed by or with the storage media switch 122 that includes the VF address engine 128 and uses the VF map 130.

[0076] Recall that at 606, the storage media switch 122 selects a virtual function of the storage media 124 for executing the host command based on the virtual function identifier. For example, the VF address engine 128 can isolate the tag portion of the address field of the host command to determine the host identifier and the virtual function identifier.

[0077] At 702, the storage media switch 122 identifies a host based on a virtual function identifier associated with the host command. The VF map 130 maintains an association or mapping between the virtual function identifier and the corresponding virtual function and the corresponding host. For example, the VF map 130 may maintain a mapping between a host identifier and a virtual function identifier to the host and the virtual function. The VF address engine 128 may determine the virtual function identifier and the host identifier from an unused portion of a virtual address included in the host command. The VF address engine 128 may determine the virtual function identifier and the host identifier from a canonical portion of a sign extension of the virtual address included in the host command.

[0078] For example, the VF address engine 128 may look up a particular host identifier (e.g., bits

[63] of the address field of the host command) from the VF map 130 to determine that the software application executing at the virtual machine VM-0 is the host associated with the particular host identifier, and therefore, the software application is the initiator of the host command. The VF address engine 128 may search the VF map 130 to determine a virtual function for the host command that matches the particular virtual function identifier (e.g., bits [62:58] of the address field of the host command).

[0079] At 704, the storage media switch 122 selects a virtual function assigned to a storage media interface of the storage media switch 122 for executing the host command based on the host and virtual function identifiers. For example, by inputting the host identifier and the virtual function identifier into the VF map 130, the VF address engine 128 can obtain a predefined internal interface routing value for engaging the appropriate virtual function.

[0080] At 706, the storage media switch 122 determines the routing address by modifying the data-location address contained in the host command. For example, the VF address engine 128 can modify the virtual address contained in the host command to append the interface selection value assigned to the virtual function to the virtual address contained in the host command. The VF address engine 128 can append the interface routing value to the front of the virtual address contained in the virtual address field of the host command to form a routing address (also referred to as a "modified virtual address"). The routing address can be different in size from the original virtual address extracted from the address field of the host command. For example, the routing address can be a 72-bit value, where bits [71:64] are the routing value and bits [63:0] are the original virtual address contained in the address field of the host command.

[0081] In some cases, the VF address engine 128 replaces any tagged or encoded portion of the routing address with a predefined value or a canonical format of a sign extension. For example, the routing address can be a 72-bit value, where bits [71:64] are the routing value, and bits [63:0] are the virtual address contained in the address field of the host command, without any unused bit tags or encodings specifying a virtual function or host identifier. The VF address engine 128 can exchange the routing value with a portion of the virtual address that is not tagged or encoded and originally contained in the virtual address to obtain a routing address for performing a read / write transaction associated with the host command. For example, the routing address can be a 72-bit value, where [71:63] is the first eight bits of the virtual address contained in the address field of the host command without any tags, the next eight bits [62:55] are the interface selection value, and bits [54:0] are the remainder of the virtual address contained in the address field of the host command without any tags. The storage media switch 122 uses the internal routing address to command the storage media 124 to perform a transaction to satisfy the host command.

[0082] At 710 , the storage media switch 122 may maintain the routing address in memory for subsequent retrieval and use after executing a transaction to satisfy the host command.

[0083] From operation 710, the storage media switch 122 may proceed to operation 608 to execute the host command based on the routing address to effectively use the selected virtual function. For example, the storage media switch may perform an operation or transaction at the region of the storage media 124 mapped to the routing address determined above.

[0084] Figure 7BAn example method 700B for responding to a host command is depicted. Method 700B is an example of operations performed by a storage media switch 122 when step 610 is performed from method 600 for automatically mapping a host command to a virtual function on a storage medium. The operations of method 700B may be performed by or with a storage media switch 122 that includes a VF address engine 128 and uses a VF map 130. In response to instructing the storage medium 124 to perform a transaction that satisfies the host command, the storage media switch 122 prepares a response to the host command by determining an original address for the host command. At 712, the storage media switch 122 removes an interface selection value from a routing address maintained in memory. For example, the VF address engine 128 may use only a portion of a 72-bit internal routing address (e.g., without an interface selection value) as an original address for a response to a host command. The VF address engine 128 may remove bits (e.g., bits [62:55]) containing an interface selection value from a routing address maintained in memory.

[0085] At 714, with the interface selection value removed, the storage media switch 122 concatenates the remaining portions of the routing address to determine the original address. For example, the storage media switch 122 may generate a 64-bit original address using the first remaining portion of the routing address (e.g., bits [71:63]) concatenated with the second remaining portion of the routing address (e.g., bits [54:0]). By removing the interface selection value (e.g., bits [62:55]) from the routing address, the original address (e.g., 64-bit) remains as bits, where bits [63:56] correspond to bits [71:63] of the routing address, and bits [55:0] correspond to bits [54:0] of the routing address.

[0086] At 716, the storage media switch 122 outputs the host command with the original address and without any virtual function tag. The storage media switch 122 may provide the original address via the host interface and to the host as part of the corresponding host command.

[0087] By executing method 700B, the storage media switch 122 can determine the original address associated with the host command for use in responding to the host command. The storage media switch 122 determines the original address by removing any virtual function tags from the virtual address included in the host command and replacing the virtual function tags with a predefined value preferred by the host or a canonical portion of the sign extension of the remainder of the virtual address.

[0088] Figure 8 is a conceptual diagram illustrating manipulation of an address field of a host command when the host command is mapped to a virtual function on a storage medium. Figure 8Example address fields 800A through 800E are included. Each of the address fields 800A through 800E is associated with a host command 802.

[0089] Figure 8 The storage media switch 122 is shown to receive a host command 802 that includes an address field 800A. The address field 800A includes an unused address portion 804A (eg, bits [63:49]) and a used address portion 806 (eg, bits [48:0]).

[0090] The storage media switch 122 can modify the host command 802 to create the address field 800B by appending the interface selection value 808 to the front of the address field 800A. The address field 800B includes the interface selection value 808 (e.g., bits [71:64]), an unused address portion 804A (e.g., bits [63:49]), and a used address portion 806 (e.g., bits [48:0]). Due to the addition of the interface selection value 808, the address field 800B is shown to have a larger size (e.g., 72 bits) than the address field 800A (e.g., 64 bits).

[0091] The storage media switch 122 may modify the unused address portion 804A of the address field 800B to remove any encoding or markings to create the address field 800C. For example, the address field 800C corresponds to the address field 800B, except that the unused address portion 804A includes different information than the unused address portion 804B. For example, the storage media switch 122 may remove any encoding or markings and replace the encoding or markings with predefined, host-specified values, or may sign-extend the canonical value associated with the used address portion 806 (e.g., by copying the value of bit

[48] to each of bits [63:49]).

[0092] The storage media switch 122 can generate an address field 800D by swapping the position of the interface selection value 808 with the position of the unused address portion 804B, which is referred to as a routing address 800D. For example, the address field 800D corresponds to the address field 800C, except that the unused address portion 804B is at bits [71:64] and the interface selection value 808 is at bits [63:49].

[0093] The storage media switch 122 may use the routing address 800D to instruct the storage media 124 to perform a transaction to satisfy the host command. Once the transaction is complete, the storage media switch 122 may respond to the host command by outputting a response that includes the original address contained in the address field 800E.

[0094] The storage media switch 122 can generate the address field 800E by removing the interface selection value 808 from the address field 800D and concatenating the untagged and unused address portion 804B with the unused address portion 806 to form a new original address. The address field 800E includes 64 bits, where bits [63:49] correspond to the untagged, unused address portion 804B, and bits [48:0] correspond to the address portion 806.

[0095] exist Figure 6-Figure 8 In some examples, storage media 124 may be SR-IOV incompatible. In other words, as described above, storage media 124 may rely on storage media switch 122 to map virtual functions on storage media 124. However, in some examples, storage media 124 may natively support SR-IOV and may have been configured to support virtual functions. In the case where storage media 124 already supports SR-IOV, storage media switch 122 may rely on native SR-IOV support to automatically map to virtual functions.

[0096] For example, the storage media switch 122 may determine a routing identifier (RID) associated with the host command. The RID may be a bus number, device number, or function number value assigned to the storage media 124. The VF address engine 128 may look up the RID at the VF map 130 to determine the host identifier and the associated virtual function identifier.

[0097] The storage media switch 122 may then perform steps 606 to 610 to perform any necessary read or write transactions to execute the host command, and then respond to the host command. For example, using the host identifier and the virtual function identifier, the storage media switch 122 may determine an interface select value from the VF map 130. The VF address engine 128 may append the interface select value to an address obtained from an unencoded or untagged host command to form a routing address. The VF address engine 128 may swap the interface select bits (e.g., bits [71:64]) with a portion of the address (e.g., bits [63:56]) to generate a routing address, which the storage media switch 122 uses to execute the host command transaction. At the end of the host command transaction, the storage media switch 122 responds to the host command by outputting the original address, which is determined by omitting the interface select bits (e.g., bits [63:56]) from the routing address to generate the original (e.g., 64-bit) address that enters with the host command.

[0098] Fig. 9An example method 900 for providing QoS for solid-state storage devices accessed through a namespace is depicted. The operations of method 900 may be performed by or with a storage media switch 122 that includes a QoS manager 132 or uses QoS parameters 134.

[0099] At 902, the storage media switch receives an I / O command for data access from a host device via a host interface. The I / O command includes an identifier for a virtual interface associated with a namespace through which data of a solid-state storage device is accessible. In some cases, the I / O command is received by a first queue of inbound command elements received from the host device, such as a VM submission queue for I / O commands. The virtual interface associated with the namespace may include an SR-IOV PCIe interface or a virtual function (VF) provided by an SR-IOV PCIe interface. In such a case, a first queue set associated with the namespace may be mapped to the VF. Alternatively or additionally, a second queue set associated with a VM executed by the host may be mapped to the VF, which effectively binds the first queue set to the second queue set.

[0100] At 904, the QoS manager determines, based on the I / O command, an amount of data of the solid-state storage device that the I / O command will access through the namespace. In some cases, the amount of data is determined based on one of a PRP list of the I / O command, a scatter-gather list of the I / O command, a number of I / O operations associated with the I / O command, or an LBA count of the I / O command.

[0101] At 906, the QoS manager determines whether the amount of data that the I / O command will access through the namespace exceeds a predefined threshold for data access through the namespace over a period of time. For example, the QoS manager may compare the number of LBAs accessed through the namespace with the LBA quota for the namespace.

[0102] Optionally, at 908, the QoS manager issues the I / O command to the solid-state storage device in response to determining that the amount of data does not exceed a predefined threshold. Alternatively, at 910, the QoS manager delays the issuance of the I / O command to the solid-state storage device in response to determining that the amount of data reaches or exceeds the predefined threshold. By doing so, the switch can provide QoS for the virtually accessed solid-state storage device based on the access parameters of the namespace.

[0103] Fig.10An example method 1000 is depicted for submitting I / O commands to a solid-state storage device based on a bandwidth quota for a namespace. The operations of method 1000 may be performed by or with a storage media switch 122 including a QoS manager 132 or using QoS parameters 134.

[0104] At 1002, the storage media switch retrieves an I / O command from a submission queue of a virtual machine mapped to a virtual function of the storage media switch. The storage media switch may be an NVMe-based storage media switch that enables mapping between a virtual function and a namespace of a solid-state storage device operably coupled to the NVMe-based storage media switch. The virtual interface associated with the namespace may include an SR-IOV PCIe interface or a virtual function (VF) provided through the SR-IOV PCIe interface.

[0105] At 1004, the QoS manager determines, based on the I / O command, an amount of data that the I / O command will access and a storage namespace to which the virtual function is mapped. The amount of data may be determined based on one of a PRP list of the I / O command, a scatter-gather list of the I / O command, a number of I / O operations associated with the I / O command, or an LBA count of the I / O command.

[0106] The QoS manager compares the amount of data that the I / O command will access to a preconfigured bandwidth quota for the storage namespace at 1006. For example, the QoS manager may compare the LBA count to an LBA threshold defined for the namespace within a certain amount of time, such as to implement bandwidth metering.

[0107] Optionally, at 1008, the QoS manager submits the I / O command to a queue of the solid-state storage device corresponding to the storage namespace. Alternatively, at 1010, the QoS manager stores the I / O command in a hierarchical queue that effectively delays the issuance of the I / O command to the solid-state storage device.

[0108] Method 1000 can proceed from operation 1010 to operation 1012, 1014 and / or operation 1016. At 1012, the QoS manager delays the issuance of I / O commands until a new timing window. Typically, whenever a new timing window begins, any I / O commands stored in the hierarchical queue can be re-evaluated for submission to a submission queue of a storage device in the storage device. At 1014, the QoS manager delays the issuance of I / O commands until an empty state is reached to the inbound queue of the QoS manager. In some cases, the inbound queue reaches an empty state during the current timing window, indicating that there are no new elements retrieved by the inbound engine that need to be evaluated for issuance. At 1016, the QoS manager delays the issuance of I / O commands until the queue of the solid-state storage device reaches an empty state. Method 1000 can return to operation 1006 from any operation in the operation to re-evaluate the I / O command for issuance to the solid-state storage device.

[0109] Fig.11 An example method 1100 for managing data access of a virtual machine through a namespace of a solid-state storage device is depicted. The operations of the method 1100 may be performed by or with a host device 102, a storage media switch 122 including a QoS manager 132 or using QoS parameters 134.

[0110] At 1102, the host associates the NVMe queue of the virtual machine to a virtual function mapped to a namespace through which the solid-state storage segment is accessible.

[0111] At 1104, the QoS manager receives, from a host on which the virtual machine is executing, parameters for QoS to be provided to data access of the virtual machine. In some cases, the QoS manager receives the parameters by which data access to the solid-state storage device through the namespace is metered or managed. In such a case, the QoS manager may determine a predefined threshold for data access through the namespace for a duration based on the parameters.

[0112] At 1106, the QoS defines a bandwidth quota for the namespace including data volume and duration based on the parameters for the QoS. At 1108, the QoS manager accesses data to the solid-state storage device by the virtual machine through the namespace to which the virtual function is mapped based on the bandwidth quota.

[0113] System on Chip

[0114] Fig.12An exemplary system on chip (SOC) 1200 is illustrated that can be implemented on a virtual interface for solid-state media (such as media accessible on an NVMe interface) or by providing virtual functions with SR-IOV to provide various aspects of QoS. SoC 1200 can be implemented in any suitable device, such as a computing device, a host device, a storage media switch, a network-attached storage device, a smart appliance, a printer, a set-top box, a server, a data center, a solid-state drive (SSD), a storage drive array, a memory module, an automotive computing system, a server, a server blade, a storage blade, a storage backplane, a storage media expansion device, a storage media card, a storage media adapter, a network-attached storage device, a network-enabled (Fabric-enabled) storage target, an NVMe-based storage controller, or any other suitable type of device (e.g., other devices described herein). Although described with reference to a SoC, Fig.12 The entity may also be implemented as other types of integrated circuits or embedded systems, such as an application specific integrated circuit (ASIC), a storage controller card, a storage backplane, a storage controller, a communication controller, an application specific standard product (ASSP), a digital signal processor (DSP), a programmable SoC (PSOC), a system-in-package (SiP), or a field programmable gate array (FPGA).

[0115] SoC 1200 may be integrated with electronic circuits, microprocessors, memory, input-output (I / O) control logic, communication interfaces, firmware, and / or software that facilitates providing functionality of a host device or storage system, such as any device or component described herein (e.g., a storage drive or storage array). SoC 1200 may also include an integrated data bus or interconnect structure (not shown) that couples various components of the SoC for data communication or routing between components. The integrated data bus, interconnect structure, or other components of SoC 1200 may be exposed or accessed through an external port, a parallel data interface, a serial data interface, a peripheral component interface, or any other suitable data interface. For example, components of SoC 1200 may access or control external storage media through an external interface or an off-chip data interface.

[0116] In this example, the SoC 1200 includes various components, such as input-output (I / O) control logic 1202 and a hardware-based processor 1204 (processor 1204), such as a microprocessor, a processor core, an application processor, a DSP, etc. (e.g., a processing resource separate from a host x86 processor). The SoC 1200 also includes a memory 1206, which may include any type and / or combination of RAM, SRAM, DRAM, non-volatile memory, ROM, one-time programmable (OTP) memory, multi-time programmable (MTP) memory, flash memory, and / or other suitable electronic data storage devices. In some aspects, the processor 1204 and the code stored on the memory 1206 are implemented as a storage media switch or a switch-enabled storage aggregator to provide various functionalities associated with providing QoS on a virtual interface for solid-state storage devices. In the context of the present disclosure, the memory 1206 stores data, code, instructions, or other information via non-transient signals and does not include carrier waves or transient signals. Alternatively or additionally, SoC 1200 may include a data interface (not shown) for accessing an additional or expandable off-chip storage medium, such as a magnetic memory or a solid-state memory (eg, flash memory or NAND memory).

[0117] The SoC 1200 may also include firmware 1208, applications, programs, software, and / or operating systems, which may be embodied as processor-executable instructions maintained on the memory 1206 for execution by the processor 1204 to implement the functionality of the SoC 1200. The SoC 1200 may also include other communication interfaces, such as a transceiver interface for controlling or communicating with components of a local on-chip (not shown) or off-chip communication transceiver. Alternatively or additionally, the transceiver interface may also include or implement a signal interface to transmit radio frequency (RF), intermediate frequency (IF), or baseband frequency signals off-chip to facilitate wired or wireless communication through a transceiver, physical layer transceiver (PHY), or media access controller (MAC) coupled to the SoC 1200. For example, the SoC 1200 may include a transceiver interface configured to enable storage over a wired or wireless network, such as to provide a network attached storage (NAS) volume with virtualized storage isolation features.

[0118] SoC 1200 also includes an instance of a storage media switch 126 (switch 126) having a VF address engine 128, a VF map 130, a QoS manager 132, and QoS parameters 134, which can be implemented separately or in combination with a storage component or data interface. According to various aspects of providing QoS on a virtual interface for solid-state storage, the switch 126 can meter or allocate bandwidth of a namespace to tenants or initiators of a host. Alternatively or additionally, the VF map 130 or the QoS parameters 134 can be stored on a memory 1206 of the SoC 1200 or on a memory operatively coupled to the SoC 1200 and accessible by the switch 126.

[0119] As described with reference to the various aspects presented herein, any of these entities may be embodied as distinct or combined components. Figure 1 A corresponding component or entity of the environment 100 or Figure 2 , Figure 3 , Figure 4 and / or Figure 5 The switch 126 may be implemented in whole or in part as processor-executable instructions maintained by the memory 1206 and executed by the processor 1204 to implement various aspects and / or features of providing QoS on a virtual interface for a solid-state storage device.

[0120] The switch 126, VF address engine 128, and / or QoS manager 132 may be implemented independently or in combination with any suitable components or circuits for implementing the various aspects described herein. For example, the VF address engine 128 and / or QoS manager 132 may be implemented as part of a DSP, a processor / memory bridge, an I / O bridge, a graphics processing unit, a memory controller, a storage controller, an arithmetic logic unit (ALU), etc. The VF address engine 128 and / or QoS manager 132 may also be provided integrated with other entities of the SoC 1200, such as with the processor 1204, memory 1206, host interface, storage medium interface, or firmware 1208 of the SoC 1200. Alternatively or additionally, the switch 126, VF address engine 128, VF map 130, QoS manager 132, and / or QoS parameters 134, and / or other components of the SoC 1200 may be implemented as hardware, firmware, fixed logic circuits, or any combination thereof.

[0121] As another example, consider Fig.13 , Fig.13An example storage media switch controller 1300 (switch controller 1300) is illustrated according to one or more aspects of providing QoS on a virtual interface for a solid-state storage device. In various aspects, the switch controller 1300 or any combination of its components can be implemented as a storage drive controller, a storage media switch, a storage media controller, a NAS controller, an NVMe initiator, an NVMe target, or a storage aggregation controller for a solid-state storage device. In some cases, the switch controller 1300 is similar to the reference 1300. Fig.12 The components of the described SoC 1200 are implemented or implemented with components of the SoC 1200. In other words, an instance of the SoC 1200 may be configured as a storage media switch controller, such as the switch controller 1300 to provide and manage QoS on a virtual interface for solid-state storage devices.

[0122] In this example, the switch controller 1300 includes input-output (I / O) control logic 1302 and a processor 1304, such as a microprocessor, a processor core, an application processor, a DSP, etc. In some aspects, the processor 1304 and firmware of the storage media switch 1300 can be implemented to provide various functionalities associated with providing QoS on a virtual interface for a solid-state storage device, such as those described with reference to methods 600, 700A, 700B, 900, 1000, and / or 1100. The switch controller also includes a storage media interface 1306 and a host interface 1308 that enable access to storage media and a host system, respectively. The storage media interface 1306 can include a physical page addressing (PPA) interface, a peripheral component interconnect express (PCIe) interface, a non-volatile memory express (NVMe) interface, a NVM-OF interface, an NVM host controller interface specification (NVMHCI) compliant interface, etc. Alternatively or additionally, the host interface may include a PCIe interface, a SATA-based interface, an NVMe interface, an NVM-OF interface, a compatible NVMHCI interface, a network-enabled storage interface, or the like.

[0123] The switch controller 1300 also includes instances of the VF address engine 128, the VF map 130, the QoS manager 132, and the QoS parameters 134. Any or all of these instances may be implemented separately as shown in the switch controller or in combination with the processor 1304, the storage media interface 1306, the host interface 1308, or the flash translation layer (not shown). Figure 1 A corresponding component or entity of the environment 100 or Figure 2 , Figure 3 , Figure 4 and / or Figure 5The switch controller 1300 may be implemented in whole or in part as processor-executable instructions maintained by a memory (not shown) of the switch and executed by a processor 1304 to implement various aspects and / or features of providing QoS on a virtual interface for a solid-state storage device.

[0124] Although the subject matter has been described in language specific to structural features and / or methodological operations, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific examples, features, or operations described herein, including orders of performance.

Claims

1. A method for providing quality of service on a virtual interface to a solid-state storage device, comprising: receiving an input / output (I / O) command for data access from a host device via a host interface, the I / O command being associated with an identifier for a virtual interface that is associated with a namespace through which data stored at the solid-state storage device is accessible, the I / O command being received via a first queue of inbound command elements received from the host device; determining, based on the I / O command, an amount of the data stored at the solid-state storage device that will be accessed by the I / O command through the namespace; determining that an amount of data to be accessed through the namespace by the I / O command exceeds a predefined threshold for data access through the namespace over a period of time; as well as In response to determining that the amount of data to be accessed by the I / O command exceeds the predefined threshold for data access through the namespace, delaying issuance of the I / O command to the solid-state storage device via a storage medium interface by storing the I / O command in a second queue for command elements that are staged for re-evaluation for issuance to the solid-state storage device, wherein the I / O command is re-evaluated in response to: (i) the first queue reaching an empty state; or (ii) a device submission queue posted to the solid-state storage device reaches an empty state.

2. The method according to claim 1, further comprising: The amount of data to be accessed by the I / O command is determined based on one of: a physical region page PRP list of the I / O command, a scatter gather list of the I / O command, a plurality of I / O operations associated with the I / O command, or a logical block address LBA count of the I / O command.

3. The method according to claim 1, further comprising: receiving, from the host device, parameters by which to meter or manage access to the data of the solid-state storage device through the namespace; as well as Based on the parameter, the predefined threshold for data accesses through the namespace within the period of time is determined. 4 . The method according to claim 1 , wherein the virtual interface associated with the namespace comprises a single root I / O virtualization (SR-IOV) peripheral component interconnect Express (PCIe) interface or a virtual function (VF) provided through a SR-IOV PCIe interface.

5. The method according to claim 4, wherein: The I / O command is received from a virtual machine VM executed by the host device; and The VM executing on the host device is mapped to the VF provided through the SR-IOV PCIe interface.

6. The method according to claim 5, wherein: A first set of queues associated with the namespace is mapped to the VF; and A second set of queues associated with the VM executed by the host is mapped to the VF, the VF effectively binding the first set of queues to the second set of queues.

7. The method according to claim 1, wherein: The host interface of the host device comprises a first PCIe interface on which a Non-Volatile Memory Express (NVMe) protocol is implemented; or The storage medium interface of the solid-state storage device includes a second PCIe interface, and the NVMe protocol is implemented on the second PCIe interface.

8. The method according to claim 1, further comprising: In response to determining that the amount of data to be accessed by the I / O command does not exceed the predefined threshold for data access through the namespace, the I / O command is issued to the solid-state storage device via the storage medium interface and based on the namespace.

9. An apparatus comprising: Hardware-based processors; a memory coupled to the processor and configured to maintain processor-executable instructions that, in response to execution, implement an application on the device; a host interface configured to enable the application to access data stored at a solid-state storage device operatively coupled to the apparatus; as well as a storage media switch comprising a storage media interface to the solid-state storage device, a first queue for inbound command elements received from the host interface, a second queue for command elements staged for subsequent issuance to the solid-state storage device, and a quality of service manager implemented as: receiving an input / output (I / O) command for data access from the application via the host interface, the I / O command being associated with an identifier of the virtual interface associated with a namespace through which data stored at the solid-state storage device is accessible, the I / O command being received through the first queue for the inbound command element; determining, based on the I / O command, an amount of the data stored at the solid-state storage device that will be accessed by the I / O command through the namespace; determining that an amount of data to be accessed through the namespace by the I / O command exceeds a predefined threshold for data access through the namespace over a period of time; as well as In response to determining that the amount of data to be accessed by the I / O command exceeds the predefined threshold for data access through the namespace, delaying issuance of the I / O command to the solid-state storage device by storing the I / O command in the second queue for the command elements that are staged for re-evaluation for issuance to the solid-state storage device, wherein the I / O command is re-evaluated in response to: (i) the first queue reaching an empty state; or (ii) a device submission queue posted to the solid-state storage device reaches an empty state.

10. The apparatus of claim 9, wherein the quality of service manager is further implemented to determine the data amount based on one of a physical region page (PRP) list of the I / O command, a scatter-gather list of the I / O command, a plurality of I / O operations associated with the I / O command, or a logical block address (LBA) count of the I / O command.

11. The apparatus according to claim 9, wherein the virtual interface associated with the namespace comprises a single root I / O virtualization (SR-IOV) peripheral component interconnect Express (PCIe) interface or a virtual function (VF) provided through the SR-IOV PCIe interface.

12. The device according to claim 11, wherein: The I / O command is received from a virtual machine VM executed by a host device; and The VM executing on the host device is mapped to the VF provided through the SR-IOV PCIe interface.

13. The apparatus of claim 9, wherein the apparatus further comprises the solid-state storage device operably coupled to the storage medium interface of the storage medium switch.

14. The device according to claim 9, wherein the quality of service manager is further implemented as: in response to determining that the amount of data to be accessed by the I / O command does not exceed the predefined threshold, issuing the I / O command to the solid-state storage device via the storage medium interface and based on the namespace.

15. A system on chip, comprising: a storage medium interface configured to support access to a solid-state storage device; a host interface configured to support communications with a host device; a first queue for inbound command elements received from the host interface; a second queue for command elements that are staged for subsequent issuance to the solid-state storage device via the storage medium interface; Hardware-based processors; a memory configured to store processor-executable instructions that, in response to execution by the hardware-based processor, implement a quality of service manager to: receiving an input / output (I / O) command for data access from the host device via the host interface, the I / O command being associated with an identifier of the virtual interface associated with a namespace through which data stored at the solid-state storage device is accessible, the I / O command being received through the first queue for the inbound command element; determining, based on the I / O command, an amount of the data stored at the solid-state storage device that will be accessed by the I / O command through the namespace; determining that an amount of data to be accessed through the namespace by the I / O command exceeds a predefined threshold for data access through the namespace over a period of time; as well as In response to determining that the amount of data to be accessed by the I / O command exceeds the predefined threshold for data access through the namespace, delaying issuance of the I / O command to the solid-state storage device by storing the I / O command in the second queue for the command elements that are staged for re-evaluation for issuance to the solid-state storage device, wherein the I / O command is re-evaluated in response to: (i) the first queue reaching an empty state; or (ii) a device submission queue posted to the solid-state storage device reaches an empty state.

16. The system on chip of claim 15, wherein the quality of service manager is further implemented to determine the data amount based on one of a physical region page (PRP) list of the I / O command, a scatter-gather list of the I / O command, a plurality of I / O operations associated with the I / O command, or a logical block address (LBA) count of the I / O command. 17 . The system on chip according to claim 15 , wherein the virtual interface associated with the namespace comprises a single root I / O virtualization (SR-IOV) peripheral component interconnect Express (PCIe) interface or a virtual function (VF) provided through the SR-IOV PCIe interface.

18. The system on chip according to claim 15, wherein: The host interface of the system on chip is configured as a first PCIe interface, on which a Non-Volatile Memory Express (NVMe) protocol is implemented; or The storage medium interface of the system on chip is configured as a second PCIe interface, and the NVMe protocol is implemented on the second PCIe interface.

19. The system on chip of claim 15, wherein the system on chip is implemented on or as a part of: a host device, a storage media switch, a server, a server blade, a storage blade, a storage backplane, a storage media expansion device, a storage media card, a storage media adapter, a network attached storage device, or an NVMe-based storage controller.

20. The system on chip of claim 15, wherein the quality of service manager is further implemented as: In response to determining that the amount of data to be accessed by the I / O command does not exceed the predefined threshold for data access through the namespace, the I / O command is issued to the solid-state storage device via the storage medium interface and based on the namespace.

Images