Enabling virtual functions on storage media

Abstract

The present disclosure describes an apparatus and method for automatically mapping virtual functions to a storage medium to enable single root input-output virtualization. A storage media switch manages access to virtual functions running behind a storage media interface managed by the switch. The switch includes a host interface where the switch receives host commands. The switch determines a virtual function identifier associated with a host command and automatically selects a virtual function of the storage medium based on the virtual function identifier. The switch uses the virtual function to execute the host command through the storage media interface and responds to each host command through the host interface after execution. By automatically mapping virtual functions in this manner, the switch automatically enables single root input-output virtualization of the storage media, including storage media that does not natively support input-output virtualization.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application claims priority to U.S. application filed in the United States on August 7, 2018 (Application No. 62 / 715,706) and U.S. application filed in the United States on August 8, 2018 (Application No. 62 / 716,278), the entire contents of which are incorporated herein by reference to the present invention. Background Technology

[0003] Generally, large-scale data computing systems are implemented as arrays of hard disk devices, solid-state memory devices, and other types of storage devices, or combinations thereof. To simplify management, reduce costs, and improve performance, some large-scale data computing systems support Single Root Input Output Virtualization (SR-IOV). Large-scale data computing systems can grow over time to include multiple generations and types of storage devices, including SR-IOV-enabled storage devices and those that do not natively support SR-IOV. Consequently, parts of large-scale data computing systems may be inefficient because common implementations of hardware-based storage virtualization cannot be utilized. The problem to be solved

[0004] delete

[0005] delete

[0006] In some aspects, a storage media switch implements a method for supporting virtual functions on a storage medium. The switch implements the method by receiving a host command from a host via a host interface and, in response to receiving the host command, determining a virtual function identifier associated with the host command. The switch also implements the method by selecting a virtual function of the storage medium to execute the host command based on the virtual function identifier. In response to executing the host command using a virtual function assigned to the storage medium, the switch proceeds with the method by responding to the host command via the host interface. By implementing a method for automatically enabling support for virtual functions in this manner, the switch can automatically (e.g., without host intervention) enable support for virtual functions on various storage media types, including storage media lacking native virtual function support. By automatically supporting virtual functions, greater isolation and partitioning capabilities can be provided to applications or services running on the host without requiring the involvement or use of host computing resources.

[0007] In other aspects, a device is disclosed, the device comprising a hardware-based processor, a memory connected to the processor and configured to maintain processor-executable instructions, and a host interface that enables the application to be implemented on the device in response to the execution of the processor-executable instructions and to act as a host for accessing data on a storage medium associated with the device. A storage medium switch of the device is connected to the host interface and provides a storage medium interface. The storage medium switch is configured to receive a host command through the host interface from the application acting as a host, and in response to receiving the host command, determine a virtual function identifier associated with the host command, and, based on the virtual function identifier, select a virtual function of the storage medium for executing the host command. In response to executing the host command through the storage medium interface and using a virtual function assigned to the storage medium, the storage medium switch is configured to respond to the host command through the host interface.

[0008] In another aspect, a system-on-chip is described as comprising a storage medium interface to a storage medium, a host interface to a host, a hardware-based processor, and memory configured to store processor-executable instructions. The processor-executable instructions implement a storage medium switch in response to execution by the hardware-based processor, and the storage medium switch receives a host command from a host through the host interface, determines a virtual function identifier associated with the host command in response to receiving the host command, and selects a virtual function of the storage medium to execute the host command based on the virtual function identifier. Additionally, the storage medium switch responds to the host command through the host interface in response to executing the host command through the storage medium interface and using a virtual function assigned to the storage medium.

[0009] Details of one or more embodiments are described in the accompanying drawings and the following description. Other features and advantages will be apparent from the detailed description, drawings, and claims. Brief explanation of the drawing

[0010] Details of one or more embodiments enabling virtual function support on a storage medium are described in the attached drawings and the following detailed description. In the drawings, the leftmost digit of a reference number indicates the drawing in which the reference number first appears. The use of the same reference number in different instances in the description and drawings indicates similar elements. FIG. 1 illustrates an exemplary operating environment having a device that enables virtual function support on a storage medium according to one or more aspects of the present invention. FIG. 2 illustrates an exemplary configuration of a storage medium switch and a solid-state drive shown in FIG. 1. FIG. 3 illustrates an exemplary configuration of a storage media switch associated with a host and a plurality of solid-state drives according to one or more aspects. FIG. 4 illustrates an exemplary configuration of various storage media switch queues according to one or more aspects. FIG. 5 illustrates an exemplary configuration of QoS parameters useful for implementing QoS according to one or more aspects. FIG. 6 illustrates an example of a method for automatically mapping host commands to virtual functions of a storage medium. FIG. 7a illustrates an exemplary method for selecting a virtual function of a storage medium. FIG. 7b illustrates an exemplary method for responding to a host command according to one or more aspects of the present invention. FIG. 8 is a conceptual diagram showing the operation of the address field of a host command when the host command is mapped to a virtual function on a storage medium. FIG. 9 illustrates an exemplary method for providing QoS to solid-state storage accessed through a namespace. FIG. 10 illustrates an exemplary method for submitting I / O commands to a solid-state storage device based on a bandwidth allocation for a namespace. FIG. 11 illustrates an exemplary method for managing data access by a virtual machine through a solid-state storage namespace. FIG. 12 illustrates an exemplary System-on-Chip (SoC) environment for implementing aspects that enable virtual functions or QoS through a virtual interface for a solid-state storage medium. FIG. 13 illustrates an exemplary storage medium switch controller for implementing aspects that enable virtual functions or QoS through a virtual interface for solid-state storage media. Specific details for implementing the invention

[0011] In large-scale data computing systems, SR-IOV can be used to manage physical storage devices as one or more virtual functions by extracting higher-layer protocols from physical connections. Through SR-IOV, the host can command virtual functions regardless of how they are implemented. Therefore, SR-IOV can partition a single storage device in various ways to make it appear as multiple virtual functions. SR-IOV can combine multiple storage devices in various ways to represent them as a single virtual function or multiple virtual functions.

[0012] Large-scale data computing systems can be implemented with arrays of storage devices that support SR-IOV and storage devices that do not support SR-IOV. Some of the storage media that do not support SR-IOV may not be available to the host of the data computing system that requires virtual functions.

[0013] As described, the storage media switch enables hosts to fully utilize virtual functions and SR-IOV capabilities by using any part of the data computing system, including using storage devices that are inherently incapable of SR-IOV. On behalf of the data computing system, the storage media switch responds to host commands transmitted to the data computing system via a standard high-speed serial computer expansion bus interface, such as PCIe® (Peripheral Component Interface Express). By managing the communication transmitted between the data computing system and the host, the storage media switch can instruct the data computing system to provide SR-IOV capabilities, including storage devices that do not natively support SR-IOV.

[0014] When virtual functions are enabled on a storage device that is essentially non-SR-IOV capable, the storage media switch automatically maps the virtual functions through a virtual address windowing scheme provided within the storage media switch. By applying the virtual address windowing scheme to host commands, the storage media switch can track host commands through a processing pipeline and cause the switch to subsequently respond to the appropriate host.

[0015] This system requires the switch to track each host command equipped with the corresponding virtual function identifier. Each unique virtual function identifier corresponds to a different virtual function. A host can assign a virtual function to a host command by including the virtual function identifier in the virtual address field of the host command. In the case of an untagged command received from a host, the switch can tag or encode the host command by automatically encoding the virtual address field of the host command with the virtual function identifier.

[0016] The portions (e.g., bits) of the virtual address field used to hold the virtual function identifier may be arbitrarily selected from any unused portion of the sign-extended canonical address or the virtual address. Internally, the switch uses these tagged address portions or bits to route host-initiated read / write transactions in response to the original host command and, upon completion, route these transactions back to the host.

[0017] The virtual address field of a host command may contain additional capacity (e.g., more bytes or bits than are required to specify a virtual function address). This additional capacity may generally not be used. For example, the virtual address field of a host command may contain information bits that are checked or not used by the host or storage media switch. The unused address portion of the virtual address field may instead contain extra information bits, for example, the sign-extended canonical portion of the virtual address contained within the host command, which may not affect the host or storage media switch if omitted or incorrect. The switch may require the unused portion of the virtual address field to include an indication of a virtual function identifier.

[0018] A host or switch may add a virtual function identifier to each host command. Before outputting host commands to the host interface of a storage media switch, the host may tag or encode the virtual function identifier to the virtual address of each host command, including the commands embedded in the physical area page (PRP) list. Alternatively, after receiving host commands from the host interface of the switch, the storage media switch may tag or encode the virtual address of each host command with the virtual function identifier, including the command PRP list.

[0019] During the execution of a host command, the switch determines a virtual function identifier encoded in the host command and initiates read or write transactions to and from host memory using the virtual function corresponding to the determined virtual function identifier. When responding to a host command, the switch may remove the virtual function identifier tagged in the virtual address of the host command. Before confirming that the host command is complete, the switch may restore parts of the virtual address of the host command that are not normally used (e.g., the sign-extended normal part of the virtual address) to a predefined state. For example, the switch may determine an "untagged" original address associated with the host command by removing the virtual function identifier from the virtual address contained within the host command and responding to the host command by including the original address and excluding the virtual function identifier (e.g., for host memory access).

[0020] When enabling virtual functions on storage devices that already inherently possess SR-IOV capabilities, the storage media switch may choose to bind front-end virtual functions to the virtual functions of the back-end storage media devices. For example, the switch can automatically map virtual functions to corresponding storage media devices by matching the drive request identifier of the storage media connected to the switch's storage media interface with the virtual functions specified in the host commands. In this way, virtual memory identifiers do not need to be tagged or encoded in the host commands to initiate read / write transactions with host memory. Rather, the virtual functions can be automatically mapped to the host specifying the virtual functions using the embedded drive request identifier.

[0021] In various aspects of automatically mapping virtual functions to storage media to enable Single Root I / O Virtualization (SR-IOV), and in various aspects of mapping virtual functions to recording media, a storage media switch manages access to virtual functions running behind the switch's storage media interface. Such a switch also includes a host interface that exchanges information with a host, including host commands. The switch determines a virtual function identifier associated with the host command and automatically selects a virtual function of the storage media based on the virtual function identifier. Alternatively, the switch selects a virtual function of the storage media based on a virtual function specified in the host command (e.g., by matching a virtual function to a drive request identifier of the storage media connected to the switch's storage media interface).

[0022] The switch uses virtual functions to execute host commands through the storage media interface and, after execution, responds to each host command through the host interface. By automatically mapping virtual functions in this manner, the switch enables Single Root I / O Virtualization (SR-IOV) across all storage media, including those that do not natively support virtual functions.

[0023] The following discussion describes an operating environment, technologies that may be used in the operating environment, and a system-on-chip (SoC) in which the components of the operating environment may be implemented. In the context of this disclosure, the operating environment is referred to merely by example.

[0024] Operating environment

[0025] FIG. 1 illustrates an exemplary operating environment (100) having host devices (102) (simply referred to as "host devices (102)"), wherein virtual function support or QoS through a virtual interface may be implemented according to one or more aspects. The host devices (102) of the operating environment (100) may store or access various forms of data, files, objects, or information. Examples of host devices (102) may include a computing cluster (104) (e.g., cloud 106), a server (108) or server hardware in a data center (110), or a server (112) (e.g., standalone), any of which may be configured as part of a storage network, storage service, or cloud system. Other examples of a host device (102) (not shown) may include a tablet computer, a set-top box, a data storage appliance, a wearable smart device, a television, a content streaming device, a High Definition Multimedia Interface (HDMI) media stick, a smart device, a home automation controller, a smart thermostat, an Internet of Things (IoT) device, a mobile internet device (MID), a Network-Attached Storage (NAS) drive, an integrated storage system, a server blade, a game console, an automotive entertainment device, an automotive computing system, an automotive control module (e.g., an engine or powertrain control module), etc. Generally, the host device (102) may communicate or store data for any suitable purpose, such as enabling the functions of a specific type of device, providing a user interface, enabling network access, implementing game applications, playing media, providing navigation, editing content, and providing a data storage.

[0026] The host device (102) includes a processor (114) and a computer-readable storage medium (116). The processor (114) may be implemented with any suitable type or number of processors (e.g., x86 or ARM) that are single-core or multi-core for executing instructions or commands of the operating system or other programs of the host device (102). The computer-readable medium (116) (CRM 116) includes a system memory (118) on which the host's virtual machine (120) may be executed or implemented. The system memory (118) of the host device (102) may include any suitable type or combination of volatile memory or non-volatile memory. For example, the volatile memory of the host device (102) may include various types of random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), etc. Non-volatile memory may include read-only memory (ROM), electronically erasable programmable ROM (EEPROM), or flash memory (e.g., NOR flash or NAND flash). These memories may be individually or in combination to store data related to applications, tenants, workloads, initiators, virtual machines, and / or operating systems of the host device (102).

[0027] In the present embodiment, the host device (102) includes a storage medium switch (122) and a storage medium (124), and the storage medium (124) can be accessed through the storage medium switch (122). Although shown as being coupled with the host device (102), the storage medium switch (122) and / or the storage medium (124) may be implemented separately from or remotely from the host device (102). The storage medium (124) of the host device (102) may be composed of any suitable type of data storage medium, such as a storage device, a storage drive, a storage array, a storage volume, etc. Although described with reference to the host device (102), the storage medium (124) may also be implemented as a standalone device or as part of a large storage aggregate such as a data center, a server farm, or a virtualized storage system (e.g., for cloud-based storage or services). Examples of storage media (124) may include a hard disk drive (HDD, not shown), an optical disk drive (not shown), a solid-state drive (126) (SSD 126), or an array of m+1 SSDs (126-0 to 126-m).

[0028] Each SSD (126) includes or is composed of a non-volatile memory device in which data or information from the host device (102) or other sources is stored. The non-volatile memory device may be implemented as any type or combination of solid-state memory media, such as flash, NAND flash, NAND memory, RAM, DRAM (e.g., for caching), SRAM, etc. In some cases, data stored in the non-volatile memory devices may be organized into data files (e.g., content) or data objects that are stored in the SSD (126) and accessed by the host device (102) or by a tenant, workload, or initiator. The type, size, or format of the files may vary depending on each source, use case, or application associated with the file. For example, files stored in the SSD (126) may include audio files, video files, text files, image files, multimedia files, spreadsheets, etc.

[0029] In the present embodiment, the storage medium switch (122) (or switch 122) of the host device (102) may support virtualization, provide QoS through a virtual interface, and implement virtual functions associated with the storage medium (124). In some aspects, the storage medium switch (122) includes a Virtual Function (VF) address engine (128), a VF mapping (130), a Quality of Service (QoS) manager (132), and QoS parameters (134), each of which may be implemented to perform the respective operation or function of supporting virtualization, providing QoS, and enabling virtual functions associated with the storage medium (124). Examples of implementations and uses of these entities are varied and are described herein.

[0030] Generally, the VF address engine (128) can enable virtual function support for storage media that inherently already support virtual functions, as well as for storage media that inherently lack SR-IOV capability. The VF address engine (128) processes a host command and automatically maps the host command to an appropriate virtual function. In some examples, the VF address engine (128) can extract a virtual function identifier from the host command. In other examples, if the storage medium (124) inherently supports SR-IOV, the VF address engine (128) can determine the virtual function identifier by searching for a routing identifier associated with the storage medium (124) within the VF mapping (130). Using the VF mapping (130), the VF address engine (128) can determine the virtual function associated with the virtual function identifier. Using the virtual function identifier, the VF address engine (128) can select the virtual function of the storage medium (124) to execute the host command. In response to instructing the storage medium (124) to execute a transaction to implement a host command, the VF address engine (128) can respond to the host command through the host interface.

[0031] In various aspects, the QoS manager (132) can ensure that the execution of a host command is managed to a quality of service level associated with a specific application, client, host, virtual machine of the host, tenant of the host, etc. For example, the QoS manager (132) can determine the QoS for a virtual machine running on the host device (102) and can ensure I / O commands and data access transactions between the host device (102) and the storage medium (124) to achieve the determined level of QoS. Alternatively or additionally, the QoS manager (132) can measure and maintain the value of the QoS parameter (134) used by the QoS manager (132) when responding to a host command.

[0032] Additionally, the host device (102) may include a data interface (140), an I / O port (136), and a graphics processing unit (138) (GPU). Generally, the I / O port (136) enables the host device (102) to interact with other devices, peripherals, or users. For example, the I / O port (136) may include or be connected to a general-purpose serial bus, a human interface device, an audio input, an audio output, etc. The GPU (138) processes and renders graphics-related data for the host device (102), such as user interface elements like an operating system or an application. In some cases, the GPU (138) accesses a portion of local memory to render graphics or includes dedicated memory (e.g., video RAM) for rendering graphics of the host device (102).

[0033] The data interface (140) of the host device (102) provides a connection to one or more networks and other devices connected to these networks. The data interface (140) may include a wired interface, such as an Ethernet or fiber optic interface, for communicating over, for example, a local network, an intranet, or the Internet. Alternatively or additionally, the data interface (140) may include a wireless interface that facilitates communication over a wireless network, such as a wireless LAN, a wide-area wireless network (e.g., a cellular network), and / or a wireless personal area network (WPAN). Any data communicated through the I / O port (136) or the data interface (140) may be written to or read from the storage medium (124) of the host device (102) according to one or more aspects of the present invention.

[0034] The data interface (140) can support the host interface of the storage media switch (122) and the storage media interface of the storage media switch (122). For example, the storage media switch (122) can receive a host command and respond to the host command from the host interface. The storage media switch (122) can instruct a transaction through the storage media interface to have the storage media (124) execute the host command.

[0035] FIG. 2 illustrates an exemplary configuration of the storage media switch (122) and SSD (126) illustrated in FIG. 1. In this example, the storage media switch (122) is operably connected between a host (202) and SSDs (126-0 to 126-n) (collectively referred to as "SSD 126"), and virtualized regions, partitions, or storage segments (e.g., isolated storage regions) are provided from the SSD (126). The storage media switch may be connected to the host (202) and / or the SSD via one or more respective PCIe interfaces (not shown) capable of communicating according to the NVMe protocol (e.g., NVMe rev 1.3). In this example, the host (202) (e.g., host device 102) includes a plurality of virtual machines (120) running on the host's computing resources (204). Generally, the computing resources (204) of the host (202) may include a combination of the processing resources and system memory of the host (202), which are used to implement applications, virtual machines, tenants, or initiators that access storage associated with the host (202).

[0036] The storage media switch (122) can enable virtual functions on the storage media and / or provide QoS through virtual functions for solid-state storage according to one or more aspects. In this example, the storage media switch (122) or its controller (not shown) enables access to the storage media associated with the switch through VFs (206-0 to 206-w) (collectively referred to as "virtual functions 206"). Generally, the virtual functions (206) focus primarily on data movement between the VM (120) of the host (202) and the storage provided by the SSD (126), and each is associated with a physical function (208). Although one example of a physical function (208) is illustrated, the storage media switch (122) may be implemented with any number and / or combination of physical functions and virtual functions. The physical function (208) of FIG. 2 may be equivalent to one or more PCI functions of a typical PCIe physical function device. In some cases, the physical function (208) is responsible for mediating policy decisions, such as link speeds or network addresses used by VMs (120) in the case of networking, and for processing various input and output transactions between the storage media switch (122), VM (120), and SSD (126).

[0037] Generally, the tenants or initiators of the VMs (120) or the host (202) access data stored on the SSD (126). In some implementations, the storage media switch (122) presents the aggregated storage media (e.g., SSD 26) to the host (202) or the host's VM (120) as a virtual disk or storage volume. In the context of NVMe, such a virtual disk may be segmented or partitioned into different regions accessed through namespaces. In other words, data access to the virtual disk or SSD (126) may be consumed through one or more namespaces corresponding to segments or stripes of the virtual disk or storage media. As illustrated in FIG. 2, each SSD (126) may be implemented with an SSD controller (210-0 to 210-n) through which it can access NAND channels (212-1 to 212-4). Each channel (212) of the NAND (e.g., channel A or NAND channel 212) includes a plurality of NAND devices, which can be implemented as separate NAND devices or NAND dies of an SSD (126) that are accessible or addressable through each NAND channel.

[0038] In some aspects, the storage media switch (122) manages access to the virtual function (206). For example, the storage media switch (122) may exchange information including host commands with the host (202), such as facilitating data access between the SSD (126) and host memory. In some cases, the storage media switch (122) determines a virtual function identifier associated with the host command and automatically selects the virtual function (206) based on the virtual function identifier. Alternatively, the switch selects the virtual function (206) based on the virtual function (206) specified in the host command, for example, by matching the virtual function (206) with the drive request identifier of the SSD (126).

[0039] Additionally, the storage media switch (122) or the processor (not shown) of the switch (122) can execute a host command of a virtual function (206) and, after execution, respond to each host command. By automatically mapping the virtual functions (206) in this way, the storage media switch (122) enables SR-IOV for any SSDs (126) (or on any SSDs (126)) including SSDs (126) that lack native support for virtual functions or SR-IOV-based features.

[0040] FIG. 3 illustrates an exemplary configuration of a storage media switch (122) (switch 122) associated with a host and a plurality of solid-state drives (collectively referred to as 300), implemented according to one or more aspects of providing QoS through a virtual interface for solid-state storage. In this example, the switch (122) is operably coupled between a host (202) (not shown) and an array of SSDs (126), each divided into eight horizontally striped namespaces (NS0 to NS31). Each of the SSDs (126) may be connected to the switch (122) via an NVMe interface, such as an NVMe interface implemented via a x4, x8, or x16 PCIe interface. Alternatively or additionally, the host (202) may be connected to the switch via an NVMe interface to enable data transactions to the SSDs (126) or other storage through the switch (122).

[0041] Generally, in this example, the NVMe specification or protocol supports namespace-based access to storage media or SSDs. In aspects of providing QoS, the virtual functions of the switch (122) are mapped to namespaces for segments or partitions of the virtual disk. By doing so, QoS using namespaces becomes possible and manageable, so that a VM or tenant can subscribe to a service level for storage access managed through namespace-based access to the storage media. In other words, a client or customer subscribes to a predefined amount of bandwidth and is connected to a virtual function by or on the host. These virtual functions mapped to namespaces within the switch (122) are allocated bandwidth or access to the storage media based on thresholds or quotas assigned to the namespaces, such as following a subscription for data access at a specific service level. Within the NVMe subsystem of the switch (122), multiple isolated domains can be supported through the use of virtual functions provided by SR-IOV over PCIe. By doing this, the administrator on the host can plug tenants into different address domains, such as VMs, and provide isolated individual access to storage, and this access and consumption of storage can take the form of namespaces.

[0042] Referring again to FIG. 3, the host (202) may be implemented as a multi-tenant host in which any suitable number of virtual machines or tenants access their respective NVMe command queues. In this example, the host (202) includes 32 VMs (not shown) each having a set of NVMe queues (302-0 to 302-31) mapped to each virtual function (304) (VF0-VF31, grouped into VF 304-0 to VF 304-31) of the switch (122). These NVMe queues (302-0 to 302-31) may be implemented in host memory and may be configured by the host to become any suitable number of administrative queues (admin queues) or I / O command queues. Here, four queues are allocated to each VM, one queue consisting of an admin queue and three queues consisting of I / O command queues.

[0043] In some aspects, the inbound and outbound queue engines of the hardware layer of the switch (122) can support up to 128 pairs of NVMe host submission queues (SQ) and host completion queues (CQ). Each of these queues can be individually mapped to any of the PCIe physical functions or virtual functions provided by the storage media switch (122). A host device or server implementing a VM can associate each host VM with each VF (304) within the switch (122). A VM can map the NVMe queue associated with the VM to each VF (304) within the switch (122) and / or bind the NVMe queue to any SQ or CQ queue provided by the switch (122). In other words, a VM manager running on a host (e.g., a server) can associate each host VM with each VF. After that, the VM itself can map an NVMe queue within that space to the VF, and thus allow the NVMe queue to be bound to any SQ or CQ provided by the switch (122).

[0044] Regarding priority, submission queues within a VM can be assigned the same, higher, or lower priority through NVMe-based arbitration policies (fixed or weighted round robin) that are available or configurable on a queue-by-queue basis. This can be very useful when one or more VMs use different submission queues among multiple submission queues to prioritize certain commands related to specific applications or workloads. In this configuration, errors and exceptions regarding VF operations can also be isolated and detected on a per-VF basis. Thus, an error occurring in one VF of the switch (122) does not interfere with the I / O command flow of other VFs of the switch. Alternatively or additionally, optional resets of the states for specific VFs, such as function-level resets, etc., are also possible.

[0045] Regarding command flow, in this implementation of the switch (122), four universal command delivery (UCD) inbound queues are mapped per VF (304). Alternatively, as long as the total queue distribution of VFs does not exceed the total number of queues supported by the switch (122) (e.g., 128 queues), the architecture of the switch (122) has the flexibility to allocate more or fewer queues per VF. Generally, the host VM will submit admin commands or I / O commands to the respective submission queues (SQ) and ring the corresponding UCD doorbell for that queue. The universal command delivery block (UCD block 306) can fetch the SQ element in response to the doorbell based on the pre-configured arbitration method of the switch (122). For example, if all I / O SQs are set to the same priority, the UCD block (306) will service these inbound queues in a round-robin manner. From the UCD block (306) or the UCD inbound queue, the switch (122) submits the fetched elements (e.g., admin or I / O commands) to a command classifier processor (CCP) or a command processor (308). The fetched elements may be submitted using the destination free list (DFL) of the I / O queue block (310) of the command processing subsystem (312) (subsystem 312) that includes or is associated with the command processor (308). Generally, the UCD block (306) will classify and distribute I / O commands (314) and admin commands (316) to corresponding queues or blocks of the subsystem (312), which also includes, in this example, an admin queue block (318) and a physical area page (PRP) list work queue block (320).

[0046] For example, considering FIG. 4, an exemplary configuration (400) of the command processing subsystem (312) is generally depicted as 400. The UCD block (306) can submit commands to the command processor (308) using a destination free list (DFL) assigned to the fetched queue. The DFL generates an inbound completion queue element (IB_CQ) for the command processor (308) based on the arbitration priority assigned to the queue. Typically, the command processor (308) or QoS manager (132) monitors the inbound completion queue for new elements and, upon receiving an element, initiates a command processing operation.

[0047] In some embodiments, the physical domain page (PRP) list task queue enables access through the VF (304) of the switch (122). For example, the VFs may use PRP data pointers, and the PRP data pointers contain virtual function (VF) identifiers inserted into the upper address bits by the processor or VF address engine for proper routing within the switch (122). Regarding command routing, I / O commands that do not include or depend on a physical domain page (PRP) list fetch (e.g., I / O commands smaller than 8 KB) will be handled by the command processor (308), so that the PRP data pointers embedded in the I / O commands can be processed before being submitted to the SSD device queue. Alternatively, I / O commands that include or depend on a physical area page (PRP) list fetch (e.g., I / O commands larger than 8 KB) are routed through a PRP list task queue to a management and execution processor (322) (MEP or management processor (322)), so that the PRP list can be processed before submitting the command to the SSD device queue.

[0048] In some aspects of providing QoS, the firmware of the command processor (308) or QoS manager (132) tracks the bandwidth used per namespace for a predefined time period. In some cases, the QoS manager (132) adds a Logical Block Address (LBA) count per I / O per namespace when a command is processed by the command processor (308). If the LBA count exceeds a predefined threshold or quota for a specific namespace, any additional commands for that namespace will be delayed or held in the respective staging queue (NSm_STAGE_Q) for each namespace. After being delayed for at least a predetermined time, the commands stored in the staging queue will be re-evaluated for submission to the SSD device submission queue (DEVn_SQ). In some cases, commands are re-evaluated when a new timing window is initiated to service the namespace queues or fetch inbound command elements. Alternatively, re-evaluation may be initiated in response to the inbound completion queue (IBn_CQ) for the command processor or the device submission queue (DEVn_SQ) reaching an empty state. In some cases, the former indicates that there are no new elements requiring evaluation fetched by the UCD block (306) during the current timing window.

[0049] To emit or submit I / O commands or elements into the submission queue of the storage device, the command processor (308) must evaluate various work queues in the following order: the namespace staging queue (NSm_STAGE_Q) corresponding to the device submission queue (DEVn_SQ), next the completion queue (MEP_CQ) of the management processor (322), and finally the inbound completion queue (IBn_CQ) of the I / O queue block or DFL queue. This order of queue priority can be reconfigured by the command processor (308) as needed or to support alternative implementations that provide QoS through the switch (122).

[0050] FIG. 5 illustrates an exemplary configuration of QoS parameters (134) useful for implementing QoS according to one or more aspects. In this example, the QoS parameters (134) are configured by reference to namespaces (502-0 to 502-31), and namespaces (502-0 to 502-31) may correspond to stripe namespaces for SSD (126) referenced in FIG. 3. Generally, the QoS parameters (134) include values ​​useful for defining or quantifying the access bandwidth provided to the namespaces, such as data amount and time period. In this example, the QoS parameters (134) include an LBA count (504), a timestamp (506), and a host submission queue consumer index (508). Alternatively or additionally, QoS parameters may include or reference a list of physical area pages (PRPs) of an I / O command (e.g., for multiple LBAs or IOPs), a scatter gather list of an I / O command (e.g., for multiple LBAs or IOPs), multiple I / O operations associated with the I / O command, or a count of logical block addresses (LBAs) of the I / O command.

[0051] Technologies for providing QoS on virtual interfaces

[0052] The following description describes a technique for providing QoS through a virtual interface for solid-state storage, which can provide storage isolation, bandwidth control, and partitioning functions to a host, tenant, or VM running on the host. These techniques may be implemented using any environment and entity described herein, such as, for example, a VF address engine (128), VF mapping (130), a QoS manager (132), or QoS parameters (134). These techniques include the methods illustrated in FIG. 6. FIGS. 7a and 7b and / or FIGS. 9 through 11 are each illustrated as a series of operations performed by one or more entities.

[0053] These methods are not necessarily limited to the order of operations depicted in the related drawings. Rather, any operations may be repeated, omitted, substituted, or rearranged to implement the various aspects described herein. Additionally, these methods may be used in whole or in part with another method, regardless of whether they are performed by the same entity, distinct entities, or any combination thereof. For example, methods may be combined to transparently provide wear leveling, load balancing, or data migration without host interaction or intervention, while exposing a virtualized isolation area of ​​the storage medium. In part of the following discussion, the operating environment (100) of FIG. 1 and the entities of FIG. 2, FIG. 3, FIG. 4 and / or FIG. 5 will be referred to as examples. Such references should be considered as exemplifying one of various examples, rather than limiting the described aspects to the operating environment (100), entities, or configurations alone. Alternatively or additionally, the operation of the method may also be implemented by or as an entity described with reference to the system-on-chip of FIG. 12 and / or the storage medium switch controller of FIG. 13.

[0054] FIG. 6 illustrates an exemplary method (600) for automatically mapping host commands to virtual functions on a storage medium. The operations of the method (600) may include a VF address engine (128) and may be performed by a storage medium switch (122) or by using a storage medium switch (122) with VF mapping (130).

[0055] In 602, the storage medium receives a host command from the host through the host interface of the storage medium switch (122). For example, the VF address engine (128) of the storage medium switch (122) can receive an indication of a new host command obtained from a host software application running on virtual machine VM-0.

[0056] In 604, in response to receiving a host command, the storage medium switch (122) determines a virtual function identifier associated with the host command. For example, the unused portion of the virtual address field of the host command may be filled with the sign-extended canonical portion of the address contained within the virtual address field. Rather than treating the entire unused portion of the address field as the sign-extended canonical portion of the address, the storage medium switch (122) may infer a virtual function identifier from the unused portion of the virtual address field and use said virtual function identifier to route the host command to be executed by the appropriate virtual function.

[0057] As an example, even if only a portion of the virtual address field, e.g., bits [48:0], is used as the virtual address of the host command, the host command may include a 64-bit virtual address field (e.g., bits [63:0]). Since a portion of the virtual address field may not be used by the storage medium switch (122) to address the host command, the unused portion of the virtual address field (e.g., bits [63:49]) may contain other information related to the host command, such as a virtual function identifier that the storage medium switch (122) can use to execute the host command.

[0058] A portion of the unused portion of the virtual address field may contain a virtual function identifier. The storage medium switch (122) may extract a virtual function identifier associated with the host command from a first unused portion of the virtual address field contained within the host command. For example, bits [62:58] may contain a virtual function identifier that the storage medium switch (122) uses to identify a specific virtual function associated with the command.

[0059] In 606, the storage medium switch (122) selects a virtual function of the storage medium (124) for executing a host command based on a virtual function identifier. For example, the VF address engine (128) may further separate the tagged portions of the address field of the host command to determine the host identifier. That is, some of the unused portions of the virtual address field may contain a host identifier, and the storage medium switch (122) may extract a host identifier associated with the host command from a second unused portion of the virtual address field contained within the host command. For example, bit

[63] may contain a host identifier that the storage medium switch (122) uses to identify a specific host associated with the command.

[0060] VF mappings (130) maintain associations or mappings between virtual function identifiers and each virtual function and each host. For example, VF mappings (130) can maintain mappings between host identifiers and virtual function identifiers for hosts and virtual functions. A VF address engine (128) that determines virtual function identifiers and host identifiers from the unused portion of a virtual address contained in a host command or from the sign-extended normal portion of a virtual address contained in a host command uses the virtual function identifiers and host identifiers to search for corresponding virtual functions using VF mappings (130).

[0061] The VF address engine (128) can search for a specific host identifier (e.g., bit

[63] ) from the VF mappings (30) to determine which host is associated with the specific host identifier, and thus determine which host is the originator of the host command. The VF address engine (128) can search the VF mappings (130) to determine the virtual function for the host command that matches the specific virtual function identifier (e.g., bits [62:58]).

[0062] In 608, the storage media switch (122) executes a host command using a virtual function assigned to the storage media (124). For example, using a host identifier and a virtual function identifier, the VF address engine (128) generates an internal interface selection value to route the host command (122) to the intended virtual function at the storage media interface of the storage media switch (122). The storage media switch (122) can ultimately route using the interface selection value and cause the storage media (124) to execute a read / write transaction to satisfy the host command.

[0063] In 610, in response to the execution of a transaction to satisfy a host command, the storage medium switch (122) responds to the host command through the host interface using a virtual function assigned to the storage medium (124). For example, in response to instructing the storage medium (124) to execute a transaction to satisfy a host command, the storage medium switch (122) prepares a response to the host command by determining the original address for the host command. The VF address engine (128) may determine the original address for the host command by using only a portion of the internal routing address (e.g., without the interface select value). The VF address engine (128) may remove the bits containing the interface select value from the routing address and use the remaining bits as the original address. For example, the original address may include bits [63:0], where bits [48:0] correspond to bits [48:0] of the original virtual address, and bits [49:63] correspond to a predefined value preferred by the host or to a regular sign extended value of the original virtual address.

[0064] The storage medium switch (122) can cause the host command to terminate the host interface using the original address without any virtual function tagging. In other words, the operations demonstrate that the storage medium switch (122) can respond to the host by determining the original address associated with the host command. The storage medium switch (122) determines the original address by removing the virtual function identifier tagged in the virtual address contained within the host command and replacing the virtual function identifier with a predefined value preferred by the host or with a sign-extended canonical part of the remaining virtual address.

[0065] FIG. 7a illustrates an exemplary method (700A) for selecting a virtual function of a storage medium to execute a host command based on a virtual function identifier determined by a relevant submission queue that has received the host command. The method (700A) is an example of operations performed by a storage medium switch (122) when executing step (606) of a method (600) for automatically mapping host commands to virtual functions on a storage medium. The operations of the method (700A) may include a VF address engine (128) and may be performed by the storage medium switch (122) or by using the storage medium switch (122) with VF mapping (130).

[0066] Looking at 606 once again, the storage medium switch (122) selects a virtual function of the storage medium (124) for executing a host command based on a virtual function identifier. For example, the VF address engine (128) can separate the tagged parts of the address field of the host command to determine the host identifier and the virtual function identifier.

[0067] In 702, the storage medium switch (122) identifies the host based on the virtual function identifier associated with the host command. The VF mapping (130) stores the association or mapping between the virtual function identifiers and each virtual function and each host. For example, the VF mapping (130) can maintain the mapping between the host identifiers and the virtual function identifiers for the hosts and virtual functions. The VF address engine (128) can determine the virtual function identifier and the host identifier from the unused portion of the virtual address included in the host command. The VF address engine (128) can determine the virtual function identifier and the host identifier from the sign-extended normal portion of the virtual address included in the host command.

[0068] For example, the VF address engine (128) can retrieve a specific host identifier (e.g., bit

[63] of the address field of the host command) from the VF mappings (130) to determine that a software application running on virtual machine VM-0 is the host associated with the specific host identifier and is therefore the originator of the host command. The VF address engine (128) can retrieve the VF mappings (130) to determine a virtual function for the host command that matches a specific virtual function identifier (e.g., bits [62:58] of the address field of the host command).

[0069] In 704, the storage media switch (122) can select a virtual function assigned to the storage media interface of the storage media switch (122) to execute a host command based on the host and virtual function identifiers. For example, by inputting the host identifier and the virtual function identifier into the VF mapping (130), the VF address engine (128) can obtain a predefined internal interface routing value to use for engaging the appropriate virtual function.

[0070] In 706, the storage medium switch (122) can determine the routing address by modifying the data-location address included in the host command. For example, the VF address engine (128) can modify the virtual address included in the host command to append the interface selection value assigned to the virtual function to the virtual address included in the host command. The VF address engine (128) can form the routing address (also referred to as the “modified virtual address”) by attaching the interface routing value to the front of the virtual address included in the virtual address field of the host command. The routing address may be of a different size from the original virtual address extracted from the address field of the host command. For example, the routing address may be a 72-bit value, where bits [71:64] are the routing value and bits [63:0] are the original virtual address included in the address field of the host command.

[0071] In some cases, the VF address engine (128) replaces any tagged or encoded portion of the routing address with a predefined value or sign-extended normal format. For example, the routing address may be a 72-bit value in which bits [71:64] are the routing value and bits [63:0] are the virtual address contained in the address field of the host command (without any tagging or encoding for unused bits specifying the virtual function or host identifier). The VF address engine (128) may exchange the routing value with the portion of the virtual address originally contained in the virtual address without tagging or encoding to derive a routing address to be used to execute a read / write transaction associated with the host command. For example, the routing address may be a 72-bit value, where bits [71:63] are the first 8 bits of the virtual address included in the address field of the host command without any tagging, the next 8 bits, bits [62:55], are the interface selection value, and bits [54:0] are the remaining bits of the virtual address included in the address field of the host command without any tagging. The storage medium switch (122) uses the internal routing address to command the storage medium (124) to execute a transaction to satisfy the host command.

[0072] In 708, the storage medium switch (122) may store routing addresses in memory for subsequent retrieval and use after executing transactions to fulfill host commands.

[0073] In operation 708, the storage media switch (122) may proceed to operation (608) to execute the host command based on the routing address to efficiently use the selected virtual function. For example, the storage media switch may perform operations or transactions in the area of ​​the storage media (124) mapped to the aforementioned routing address.

[0074] FIG. 7b illustrates an exemplary method (700B) for responding to a host command. Method (700B) is an example of operations performed by a storage medium switch (122) when executing step (610) of a method (600) for automatically mapping host commands to virtual functions on a storage medium. The operations of method (700B) may be performed by the storage medium switch (122) or by using the storage medium switch (122) with VF address engine (128) and VF mapping (130). In response to instructing the storage medium (124) to execute transactions satisfying the host command, the storage medium switch (122) prepares a response to the host command by determining the original address for the host command. In 712, the storage medium switch (122) removes the interface selection value from the routing address stored in memory. For example, the VF address engine (128) may use only a portion of the 72-bit internal routing address (e.g., without the interface select value) as the original address for the response to the host command. The VF address engine (128) may remove the bits containing the interface select value (e.g., bits [62:55]) from the routing address stored in memory.

[0075] In 714, the storage medium switch (122) determines the original address by concatenating the removed interface selection value and the remainder of the routing address. For example, the storage medium switch (122) can generate a 64-bit original address using the first remainder of the routing address (e.g., bits [71:63]) concatenated with the second remainder of the routing address (e.g., bits [54:0]). By removing the interface selection value (e.g., bits [62:55]) from the routing address, the original address (e.g., 64-bit) is left with bits [63:56] corresponding to bits [71:63] of the routing address and bits [55:0] corresponding to bits [54:0] of the routing address.

[0076] In 716, the storage medium switch (122) outputs a host command to the original address without any virtual function tag. The storage medium switch (122) may provide the original address to the host through the host interface as part of responding to the host command.

[0077] By performing the above method (700B), the storage medium switch (122) can determine the original address associated with the host command to be used in response to the host command. The storage medium switch (122) can determine the original address by removing any virtual function tagging from the virtual address contained within the host command and replacing the virtual function tagging with a host preferred predefined value or a sign-extended normal part of the remaining virtual address.

[0078] FIG. 8 is a conceptual diagram illustrating the operation of the address fields of a host command when the host command is mapped to virtual functions on a storage medium. FIG. 8 includes exemplary address fields (800A to 800E). Each of the address fields (800A to 800E) is associated with a host command (802).

[0079] FIG. 8 indicates that a storage medium switch (122) can receive a host command (802) containing an address field (800A). The address field (800A) includes an unused address portion (804A) (e.g., bits [63:49]) and a used address portion (806) (e.g., bits [48:0]).

[0080] The storage medium switch (122) can create an address field (800B) by modifying the host command (802) by attaching an interface selection value (808) to the front of the address field (800A). The address field (800B) includes an interface selection value (808) (e.g., bits [71:64]), an unused address portion (804A) (e.g., bits [63:49]), and a used address portion (806) (e.g., bits [48:0]). Due to the addition of the interface selection value (808), the address field (800B) is shown to have a larger size (e.g., 72 bits) than the address field (800A) (e.g., 64 bits).

[0081] The storage medium switch (122) may modify the unused address portion (804A) of the address field (800B) to create an address field (800C) by removing any encoding or tagging. For example, the address field (800C) corresponds to the address field (800B) except that the unused address portion (804A) contains different information from the unused address portion (804B). For example, the storage medium switch (122) may remove any encoding or tagging and replace said encoding or tagging with a predefined, host-specified value, or sign-extend the normal value associated with the used address portion (806) (for example, by duplicating the value of bit

[48] to each of the bits [63:49]).

[0082] The storage medium switch (122) can generate an address field (800D) referred to as a routing address (800D), by exchanging the location of the interface selection value (808) with the location of the unused address portion (804B). For example, the address field (800D) corresponds to the address field (800C), except that the unused address portion (804B) is in bits [71:64] and the interface selection value (808) is in bits [63:49].

[0083] The storage medium switch (122) may use the routing address (800D) to instruct the storage medium (124) to execute a transaction for the fulfillment of a host command. When the transaction is completed, the storage medium switch (122) may respond to the host command by outputting a response containing the original address contained in the address field (800E).

[0084] The storage medium switch (122) can create an address field (800E) by removing the interface selection value (808) from the address field (800D) to form a new original address and connecting the untagged and unused address portion (804B) with the used address portion (806). The address field (800E) contains 64 bits, where bits [63:49] correspond to the untagged and unused address portion (804B) and bits [48:0] correspond to the address portion (806).

[0085] In the examples of FIG. 6-8, the storage medium (124) may not be compatible with SR-IOV. That is to say, the storage medium (124) may rely on the storage medium switch (122) to map virtual functions on the storage medium (124) as described above. However, in some examples, the storage medium (124) may natively support SR-IOV and may already be configured to support virtual functions. If the storage medium (124) already supports SR-IOV, the storage medium switch (122) may automatically map to virtual functions by relying on native SR-IOV support.

[0086] For example, the storage medium switch (122) can determine a routing identifier (RID) associated with a host command. The RID may be a bus number, device number, or function number value assigned to the storage medium (124). The VF address engine (128) can determine the host identifier and the associated virtual function identifier by retrieving the RID from the VF mapping (130).

[0087] The storage medium switch (122) can execute and respond to a host command by executing steps 606 through 610, which execute any necessary read or write transactions. For example, using a host identifier and a virtual function identifier, the storage medium switch (122) can determine an interface selection value from the VF mapping (130). The VF address engine (128) can form a routing address by adding the interface selection value to an address derived from an unencoded or untagged host command. The VF address engine (128) can exchange interface selection bits (e.g., bits [71:64]) with a part of an address (e.g., bits [63:56]) to generate a routing address, and the storage medium switch (122) uses this routing address to execute a host command transaction. When the host command transaction is completed, the storage medium switch (122) can respond to the host command by outputting the original address determined by omitting the interface selection bits (e.g., bits [63:56]) from the routing address to generate the original (e.g., 64-bit) address provided with the host command.

[0088] FIG. 9 illustrates an exemplary method (900) for providing QoS for solid-state storage accessed through a namespace. The operations of the method (900) may be performed by a storage medium switch (122) or by means of a QoS manager (132) or by means of a QoS parameter (132).

[0089] In 902, the storage media switch receives I / O commands for data access from a host device through a host interface. The I / O commands include identifiers for virtual interfaces associated with a namespace where data in solid-state storage can be accessed. In some cases, the I / O commands are received through a first queue of inbound command elements received from the host device, such as a VM submission queue for I / O commands. The virtual interface associated with the namespace may include an SR-IOV PCIe interface or a virtual function (VF) provided through the SR-IOV PCIe interface. In this case, a first set of queues associated with the namespace may be mapped to the VF. Alternatively, or additionally, a second set of queues associated with a VM running on the host may be mapped to the VF, so that the first set of queues can be effectively bound to the second set of queues.

[0090] In 904, the QoS manager determines, based on the I / O command, the amount of solid-state data that the I / O command accesses through the namespace. In some cases, the amount of data is determined based on one of the I / O command's PRP list, the I / O command's scatter gather list, the number of I / O operations associated with the I / O command, or the I / O command's LBA count.

[0091] In 906, the QoS manager determines whether the amount of data an I / O command accesses through a namespace exceeds a predefined threshold (a threshold for data access through a namespace for a given period of time). For example, the QoS manager can compare the number of LBAs accessed through a namespace with the LBA quota for that namespace.

[0092] Optionally, at 908, the QoS manager releases an I / O command to solid-state storage in response to determining that the amount of data does not exceed a predefined threshold. Alternatively, at 910, the QoS manager delays the release of an I / O command to solid-state storage in response to determining that the amount of data satisfies or exceeds a predefined threshold. By doing so, the switch can provide QoS for virtually accessed solid-state storage based on the access parameters of the namespace.

[0093] FIG. 10 illustrates an exemplary method (1000) for submitting an I / O command to a solid-state storage device based on a bandwidth quota of a namespace. The operations of the method (1000) may be performed by a storage medium switch (122) or by using a storage medium switch (122), including a QoS manager (132) or using QoS parameters (132).

[0094] In 1002, the storage media switch fetches I / O commands from the submission queue of a virtual machine mapped to a virtual function of the storage media switch. The storage media switch may be an NVMe-based storage media switch that enables mapping between namespaces and virtual functions of solid-state storage operablely connected to the NVMe-based storage media switch. A virtual interface associated with a namespace may include an SR-IOV PCIe interface or a virtual function (VF) provided through the SR-IOV PCIe interface.

[0095] In 1004, the QoS manager determines, based on the I / O command, the amount of data the I / O command will access and the storage namespace to which the virtual function is mapped. The amount of data may be determined based on one of the I / O command's PRP list, the I / O command's scatter gather list, the number of I / O operations associated with the I / O command, or the I / O command's LBA count.

[0096] In 1006, the QoS manager compares the preset bandwidth quota for the storage namespace with the amount of data that the I / O command will access. For example, to implement bandwidth metering, the QoS manager can compare the LBA count with the LBA threshold defined for the namespace over a specific period of time.

[0097] Optionally, at 1008, the QoS manager submits the I / O command to the queue of the solid-state storage device corresponding to the storage namespace. Alternatively, at 1010, the QoS manager stores the I / O command in a staging queue to effectively delay the release of the I / O command to solid-state storage.

[0098] From operation (1010), the method (1000) may proceed to operations (1012, 1014 and / or 1016). In 1012, the QoS manager delays the release of I / O commands until a new timing window. Generally, any I / O commands stored in the staging queue may be re-evaluated to be submitted to the submission queue of one of the storage devices whenever a new timing window begins. In 1014, the QoS manager delays the release of I / O commands until the inbound queue for the QoS manager reaches an empty state. In some cases, the inbound queue reaches an empty state in the current timing window, indicating that there are no new elements to be fetched by the inbound engine and evaluated for release. In 1016, the QoS manager delays the release of I / O commands until the queue of the solid-state storage device reaches an empty state. From any operation, the method (1000) can return to operation (1006) and re-evaluate the I / O command for release to a solid-state storage device.

[0099] FIG. 11 illustrates an exemplary method (1100) for managing data access by a virtual machine through a namespace of solid-state storage. The operations of the method (1100) may be performed by or using a host device (102), a storage medium switch (122), including a QoS manager (132) or using QoS parameters (132).

[0100] In 1102, the host associates the NVMe queues of the virtual machine with a virtual function, which is mapped to a namespace where a segment of solid-state storage can be accessed.

[0101] In 1104, the QoS manager receives parameters for the QoS of data access to be provided to the virtual machine from the host where the virtual machine is running. In some cases, the QoS manager receives parameters that can measure or manage data access to solid-state storage through a namespace. In such cases, the QoS manager can determine a predefined threshold for data access through the namespace for a specified period of time based on the parameters.

[0102] In 1106, QoS defines a bandwidth quota for a namespace, including data volume and time period, based on parameters for the namespace and QoS. In 1108, the QoS manager measures data access to solid-state storage by virtual machines based on the bandwidth quota, and this data access is performed through the namespace to which the virtual function is mapped.

[0103] System-on-Chip

[0104] FIG. 12 illustrates an exemplary System-on-Chip (SoC) (1200) capable of implementing various aspects of providing QoS, such as through an NVMe interface or through a virtual interface to a solid-state medium accessible via a virtual function provided by SR-IOV. The SoC (1200) may be implemented as a suitable device, such as a computing device, a host device, a storage media switch, network-attached storage, a smart device, a printer, a set-top box, a server, a data center, a Solid-State Drive (SSD), a storage drive array, a memory module, an automotive computing system, a server, a server blade, a storage blade, a storage backplane, a storage media expansion device, a storage media card, a storage media adapter, a fabric-supported storage target, an NVMe-based storage controller, or other suitable types of devices (e.g., others described herein). Although described with reference to an SoC, the entity of FIG. 12 may also be implemented as other types of integrated circuits or embedded systems, such as an application-specific semiconductor (ASIC), a storage controller card, a storage backplane, a storage controller, a communication controller, an application-specific standard product (ASSP), a digital signal processor (DSP), a programmable SoC (PSoC), a system-in-package (SiP), or a field-programmable gate array (FPGA).

[0105] The SoC (1200) may be integrated with electronic circuits, a microprocessor, memory, input / output (I / O) control logic, communication interfaces, and firmware, and / or may be integrated with software useful for providing functions of a host device or storage system, such as any devices or components described herein (e.g., a storage drive or a storage array). Additionally, the SoC (1200) may include an integrated data bus or interconnection fabric (not shown) connecting various components of the SoC for data communication or routing between components. The integrated data bus, interconnection fabric, or other components of the SoC (1200) may be exposed or accessed through external ports, parallel data interfaces, serial data interfaces, peripheral component interfaces, or other suitable data interfaces. For example, components of the SoC (1200) may access or control external storage media through external interfaces or off-chip data interfaces.

[0106] In this example, the SoC (900) may include various components, such as input / output (I / O) control logic (1202) and a hardware-based processor (1204) (processor 1204), such as a microprocessor, processor core, application processor, DSP, etc. (e.g., processing resources separate from the host x86 processor). Additionally, the SoC (1200) may include memory (1206), which may include any type and combination of RAM, SRAM, DRAM, non-volatile memory, ROM, one-time programmable (OTP) memory, multiple-time programmable (MTP) memory, flash memory, and / or other suitable electronic data storage devices. In some aspects, the code and processor (1204) stored in the memory (1206) are implemented as a storage medium switch or a switchable storage aggregator, providing various functions related to providing QoS through a virtual interface to solid-state storage. In the content of this specification, the memory (1206) stores data, code, instructions, or other information via non-transient signals and does not include carrier waves or transient signals. Alternatively or additionally, the SoC (1200) may include a data interface (not shown) for accessing additional or expandable off-chip storage media, such as magnetic memory or solid-state memory (e.g., flash or NAND memory).

[0107] Additionally, the SoC (1200) may include firmware (1208), applications, programs, software, and / or operating systems, which may be implemented as processor-executable instructions maintained in memory (1206) for execution by the processor (1204) to implement the functions of the SoC (1200). Additionally, the SoC (1200) may include other communication interfaces, such as a transceiver interface for controlling or communicating with components of a local on-chip (not shown) or off-chip communication transceiver. Alternatively or additionally, the transceiver interface may include or implement a signal interface for communicating radio frequency (RF), intermediate frequency (IF), or baseband frequency signals to facilitate wired or wireless communication through a transceiver, a physical layer transceiver (PHY), or a media access controller (MAC) connected to the SoC (1200). For example, the SoC (1200) may include a transceiver interface that enables storage over a wired or wireless network to provide, for instance, a network attached storage (NAS) volume equipped with virtualized storage separation features.

[0108] Additionally, the SoC (1200) includes an example of a storage medium switch (126) equipped with a VF address engine (128), VF mapping (130), QoS manager (132), and QoS parameters (134), which may be implemented individually as illustrated or combined with a storage component or data interface. Depending on the various aspects of providing QoS through a virtual interface to solid-state storage, the switch (126) may measure the bandwidth of a namespace or allocate it to a tenant or initiator of the host. Alternatively or additionally, the VF mapping (130) or QoS parameters (134) may be stored in memory (1206) of the SoC (1200) or in memory operablely coupled with the SoC (1200) and accessible to the switch (126).

[0109] Any of these entities may be implemented as separate components or combined components, as described with reference to the various aspects disclosed herein. Examples of these components and / or entities, or corresponding functions, are described with reference to each component or entity of the environment (100) of FIG. 1 or each configuration shown in FIG. 2 through 5. All or part of the switch (126) may be implemented as processor-executable instructions maintained by memory (1206) and executed by the processor (1204), thereby implementing various aspects and / or features that provide QoS through a virtual interface to solid-state storage.

[0110] The switch (126), VF address engine (128), and / or QoS manager (132) may be implemented independently or in combination with any suitable component or circuit to implement the aspects described herein. For example, the VF address engine (128) and / or QoS manager (132) may be implemented as part of a DSP, a processor / storage bridge, an I / O bridge, a graphics processing unit, a memory controller, a storage controller, an arithmetic logic unit (ALU), etc. Additionally, the VF address engine (128) and / or QoS manager (132) may be provided integrated with other entities of the SoC (1200), such as the processor (1204), memory (1206), host interface, storage media interface, or firmware (1208) of the SoC (1200). Alternatively or additionally, the switch (126), VF address engine (128), VF mapping (130), QoS manager (132) and / or QoS parameter (134) and / or other components of the SoC (1200) may be implemented as hardware, firmware, fixed logic circuits, or any combination thereof.

[0111] As another example, referring to FIG. 13, FIG. 13 illustrates an exemplary storage media switch controller (1300) (switch controller 1300) according to one or more aspects of providing QoS through a virtual interface to solid-state storage. In various aspects, any combination of the switch controller (1300) or the components of the switch controller (1300) may be implemented as a storage drive controller, a storage media switch, a storage media controller, a NAS controller, an NVMe initiator, an NVMe target, or a storage aggregation controller for solid-state storage. In some cases, the switch controller (1300) is implemented similarly to that described with reference to FIG. 12 or together with the components of the SoC (1200). That is, an instance of the SoC (1200) may be configured as a storage media switch controller, such as the switch controller (1300), which provides and manages QoS through a virtual interface to solid-state storage.

[0112] In this embodiment, the switch controller (1300) includes input / output (I / O) control logic (1302) and a processor (1304), such as a microprocessor, processor core, application processor, DSP, etc. In some aspects, the processor (1304) and firmware of the storage medium switch (1300) may be implemented to provide various functions related to providing QoS through a virtual interface to solid-state storage, as described with reference to methods (600, 700A, 700B, 900, 1000 and / or 1100). Additionally, the switch controller includes a storage medium interface (1306) and a host interface (1308), which allow access to the storage medium and the host system, respectively. The storage media interface (1306) may include a Physical Page Addressing (PPA) interface, a Peripheral Component Interconnect Express (PCIe) interface, a Non-Volatile Memory Express (NVMe) interface, an NVM over Fabric interface, an NVM Host Controller Interface Specification (NVMHCIS) compatible interface, etc. Alternatively or additionally, the host interface may include a PCIe interface, a SATA-based interface, an NVMe interface, an NVM-OF interface, an NVMHCIS compatible interface, a fabric-enabled storage interface, etc.

[0113] Additionally, the switch controller (1300) includes instances of a VF address engine (128), VF mapping (130), QoS manager (132), and QoS parameters (134). All or part of these may be implemented individually as illustrated in the switch controller, or may be implemented in combination with a processor (1304), a storage media interface (1306), a host interface (1308), or a flash conversion layer (not illustrated). Examples of these components and / or entities, or corresponding functions, are described with reference to each component or entity of the environment (100) of FIG. 1 or each configuration illustrated in FIG. 2 through 5. All or part of the switch controller (1300) may be implemented as processor-executable instructions maintained by the switch's memory (not illustrated) and executed by the processor (1304), thereby implementing various aspects and / or features of providing QoS through a virtual interface to solid-state storage.

[0114] Although the subject matter of the invention has been described in language specific to structural features and / or methodological operations, it should be noted that the subject matter of the invention as described in the claims, including the order in which they are performed, is not necessarily limited to these specific examples, features, or operations alone.

Claims

Claim 1 A method performed by a storage medium switch to support Single Root Input Output Virtualization (SR-IOV) on a storage medium so that the storage medium appears as multiple virtual functions, the method comprising: receiving a host command from a host through a host interface of the storage medium switch (602); determining a virtual function identifier associated with the host command in response to receiving the host command (604); selecting one virtual function among the virtual functions associated with the storage medium based on the virtual function identifier to execute the host command (606), wherein the step of selecting the virtual function (606) comprises identifying a host based on the host command (702) and selecting a virtual function assigned to the storage medium interface of the storage medium switch based on the host and virtual function identifier to execute the host command (704); and generating a routing address for the host command by modifying a virtual address included in the host command (706), wherein the virtual address is an interface selection value assigned to the selected virtual function in the virtual address A method performed by a storage medium switch, characterized by comprising: a step (710) of maintaining the routing address in memory by modifying it by appending; a step (608) of executing the host command to perform an operation in an area of ​​the storage medium associated with the routing address; a step (712) of determining the original address associated with the host by removing the interface selection value from the routing address maintained in memory in response to executing the host command (608) using a selected virtual function; and a step (610) of providing a response to the host command using the original address associated with the host through the host interface. Claim 2 A method performed by a storage medium switch according to claim 1, wherein a virtual function identifier associated with the host command is determined based on a virtual address included in the host command. Claim 3 A method performed by a storage medium switch according to paragraph 2, wherein the unused portion of the virtual address included in the host command includes a virtual function identifier associated with the host. Claim 4 A method performed by a storage medium switch according to paragraph 2, wherein the sign-extended canonical portion of a virtual address included in the host command includes a virtual function identifier associated with the host command. Claim 5 A device comprising: a hardware-based processor; a memory connected to the processor and configured to maintain processor-executable instructions, wherein the processor-executable instructions implement an application on the device in response to execution; a host interface that enables the application to act as a host for accessing data of a storage medium associated with the device; and a storage medium switch connected to the host interface and providing a storage medium interface, wherein the storage medium switch is configured to perform the method of any one of claims 1 to 4. Claim 6 A device according to claim 5, characterized in that the host interface includes a Peripheral Component Interface Express (PCIe) bus interface. Claim 7 In paragraph 5, the device is characterized by being implemented as a system-on-chip. Claim 8 A device according to claim 7, wherein the system-on-chip is implemented as part of a host device, server, server blade, storage blade, storage backplane, storage media expansion device, storage media card, storage media adapter, network attached storage, fabric-enabled storage target, or storage controller. Claim 9 delete Claim 10 delete Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images