Quality of service control for logic devices of memory subsystems

Abstract

The invention relates to quality of service control for a logic device of a memory subsystem. A processing device in a memory subsystem iteratively processes input / output (I / O) operations corresponding to a plurality of logic devices associated with a memory device. For each of the plurality of logical devices, the processing includes identifying a current logical device, determining one or more I / O operations in a queue for the current logical device, and determining a number of operation credits associated with the current logical device. The number of credits is based at least in part on a set of quality of service (QoS) parameters for the current logical device. The processing further includes, in response to determining that the number of operational credits satisfies a threshold condition, performing the one or more I / O operations for the current logic device and identifying a subsequent logic device of the plurality of logic devices.

Description

[0001] Divisional application

[0002] This application is a divisional application of the invention patent application with application date of December 31, 2020, application number 202080093578.X, and invention name “Quality of Service Control of Logical Devices for Memory Subsystems”. Technical Field

[0003] The present disclosure relates generally to memory subsystems, and more particularly, to quality of service control for logic devices of a memory subsystem. Background Art

[0004] The memory subsystem may include one or more memory devices that store data. The memory devices may be, for example, non-volatile memory devices and volatile memory devices. In general, the host system may utilize the memory subsystem to store data at the memory devices and retrieve data from the memory devices. Summary of the invention

[0005] According to one aspect of the present disclosure, a method is provided. The method includes: iteratively processing input / output (I / O) operations corresponding to a plurality of logical devices associated with a memory device of a memory subsystem, wherein for each of the plurality of logical devices, the processing includes: identifying, by a processing device, a current logical device; determining one or more I / O operations in a queue for the current logical device; determining a number of operation credits associated with the current logical device, wherein the number of credits is based at least in part on a set of quality of service (QoS) parameters for the current logical device; in response to determining that the number of operation credits satisfies a threshold condition, executing, by the processing device, the one or more I / O operations for the current logical device; and identifying a subsequent logical device among the plurality of logical devices.

[0006] According to another aspect of the present disclosure, a system is provided. The system includes: a memory device; and a processing device, which is operatively coupled to the memory device, the processing device performing operations including: iteratively processing input / output (I / O) operations corresponding to a plurality of logical devices associated with the memory device, wherein for each of the plurality of logical devices, the processing includes: identifying, by the processing device, a current logical device; determining one or more I / O operations in a queue for the current logical device; determining a number of operation credits associated with the current logical device, wherein the number of credits is based at least in part on a set of quality of service (QoS) parameters for the current logical device; in response to determining that the number of operation credits satisfies a threshold condition, executing, by the processing device, the one or more I / O operations for the current logical device; and identifying a subsequent logical device among the plurality of logical devices.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium includes instructions that, when executed by a processing device, cause the processing device to perform operations including: providing multiple physical functions at a memory device; presenting the multiple physical functions to a host computing system through a peripheral component interconnect express (PCIe) interface, wherein each of the multiple physical functions corresponds to a different virtual machine running on the host computing system; and managing input / output (IO) operations received from the host computing system and for the multiple physical functions in view of a number of operation credits associated with each physical function, wherein the number of credits is based at least in part on a set of quality of service (QoS) parameters for the corresponding physical function. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present disclosure will be more fully understood from the detailed description given below and the accompanying drawings of various embodiments of the present disclosure.

[0009] Figure 1 An example computing system including a memory subsystem according to some embodiments of the present disclosure is described.

[0010] Figure 2 An example interface between a host system and a memory subsystem controller implementing a quality of service (QoS) management component according to some embodiments of the present disclosure is described.

[0011] Figure 3 An example of credit management for a logical device according to some embodiments of the present disclosure is described.

[0012] Figure 4An example physical host interface between a host system and a memory subsystem implementing quality of service management according to some embodiments of the present disclosure is described.

[0013] Figure 5 A method for quality of service control of a logic device of a memory subsystem according to some embodiments of the present disclosure is described.

[0014] Figure 6 Methods of processing I / O operations for a logic device of a memory device in view of quality of service parameters according to some embodiments of the present disclosure are described.

[0015] Figure 7 is a block diagram of an example computer system in which embodiments of the present disclosure may operate. DETAILED DESCRIPTION

[0016] Aspects of the present disclosure are directed to quality of service control of a logical device in a memory subsystem. The memory subsystem may be a storage device, a memory module, or a mixture of a storage device and a memory module. Figure 1 Examples of storage devices and memory modules are described. In general, a host system may utilize a memory subsystem that includes one or more components, such as memory devices that store data. The host system may provide data for storage at the memory subsystem and may request retrieval of data from the memory subsystem.

[0017] The memory subsystem may include high-performance memory devices that can be partitioned across multiple clients or users. Conventional memory subsystems configured in this manner may partition one or more memory devices into logical devices that may be assigned to different users or for different application purposes. In such examples, the total capacity of the physical device as well as the access bandwidth and input / output operations per second (IOPS) capabilities may be apportioned to the logical devices (sometimes referred to as virtualized "physical functions"), which are evenly partitioned across the logical devices or partitioned with different capacity profiles or configurations based on specific client or customer implementations. For example, a 1 terabyte (Tb) physical device may be partitioned into multiple logical devices with equivalent capacity (e.g., 10 100-gigabyte (Gb) partitions, 1000 1Gb partitions, etc.) or different capacities (e.g., 5 100Gb partitions and 20 25Gb partitions). In addition, in addition to or in lieu of storage capacity, available access bandwidth and / or IOPS capabilities may also be partitioned between multiple logical devices. For example, each logical device may be assigned a certain percentage of available bandwidth and / or IOPS capability.

[0018] Conventional memory subsystems configured in this way usually implement quality of service (QoS) policies to provide users with performance agreements for each logical device or partition. In these cases, QoS policies usually provide latency protocols to ensure that read and / or write operations meet predetermined thresholds. However, in many conventional implementations, QoS policies are evenly applied across partitions of memory devices. In other words, the latency capabilities of physical devices are evenly distributed across logical devices / partitions. This can significantly limit the flexibility of implementation of logical devices because it hinders the configuration of high-performance partitions and lower-performance partitions on the same physical drive. In addition, many conventional memory subsystems implement QoS policies in terms of latency rather than bandwidth measurement or IOPS measurement. In addition, conventional implementations are configured so that the QoS policy of a logical device usually does not operate with any perception of other logical devices for the same physical device. In such instances, when one logical device is idle and other logical devices are experiencing heavier workloads, the total bandwidth of the physical device is often underutilized.

[0019] Aspects of the present disclosure address the above and other deficiencies by implementing a QoS manager to facilitate quality of service control of logical devices in a memory subsystem. The QoS manager may receive bandwidth QoS parameters associated with logical device partitions for a physical device and divide the device bandwidth into sub-bands, where each sub-band is associated with one of the logical devices. The QoS manager may then determine I / O operations in a queue for a particular logical device and perform the queued operations using an earned credit scheme, where the logical device earns credits based on the QoS parameters. In addition, the QoS manager may operate with awareness of all logical devices associated with the physical device, such that any underutilized bandwidth of one logical device may be redirected to another logical device during periods of more efficient workloads.

[0020] Advantages of the present disclosure include, but are not limited to, significantly improved bandwidth performance for logical devices that are split across physical devices in a memory subsystem. A QoS manager implementing the present disclosure can ensure QoS IOPS and bandwidth agreements for each logical device within its own performance subband using a credit scheme adjusted according to QoS parameters. In addition, because the QoS manager of the present disclosure is aware of the QoS parameters and current performance of each of the logical devices, any underutilized capacity of the physical devices can be redirected to any logical devices that are operating under a heavier workload. Furthermore, because the bandwidth performance of the memory devices is improved, the performance of the memory subsystem that includes the memory devices is also improved because fewer I / O bottlenecks are encountered across the memory subsystem.

[0021] Figure 1An example computing system 100 is illustrated that includes a memory subsystem 110 according to some embodiments of the present disclosure. Memory subsystem 110 may include media such as one or more volatile memory devices (e.g., storage device 140), one or more non-volatile memory devices (e.g., memory device 130), or a combination of the like.

[0022] The memory subsystem 110 may be a storage device, a memory module, or a mixture of storage devices and memory modules. Examples of storage devices include solid-state drives (SSDs), flash drives, universal serial bus (USB) flash drives, secure digital (SD) cards, embedded multimedia controller (eMMC) drives, universal flash memory (UFS) drives, and hard disk drives (HDDs). Examples of memory modules include dual in-line memory modules (DIMMs), small outline DIMMs (SO-DIMMs), and various types of non-volatile dual in-line memory modules (NVDIMMs).

[0023] The computing system 100 can be a computing device, such as a desktop computer, a laptop computer, a network server, a mobile device, a vehicle (e.g., an airplane, drone, train, car, or other transportation vehicle), an Internet of Things (IoT)-enabled device, an embedded computer (e.g., a computer included in a vehicle, industrial equipment, or a networked commercial device), or such computing device that includes a memory and a processing device.

[0024] The computing system 100 may include a host system 120 coupled to one or more memory subsystems 110. In some embodiments, the host system 120 is coupled to memory subsystems 110 of different types. Figure 1 An example of a host system 120 coupled to one memory subsystem 110 is illustrated. As used herein, "coupled to" or "coupled with" generally refers to a connection between components, which may be an indirect communication connection or a direct communication connection (e.g., without intermediate components), whether wired or wireless, including connections such as electrical connections, optical connections, magnetic connections, etc.

[0025] The host system 120 may include a processor chipset and a software stack executed by the processor chipset. The processor chipset may include one or more cores, one or more caches, a memory controller (e.g., an NVDIMM controller), and a storage protocol controller (e.g., a PCIe controller, a SATA controller). The host system 120 uses, for example, the memory subsystem 110 to write data to the memory subsystem 110 and read data from the memory subsystem 110.

[0026] The host system 120 may be coupled to the memory subsystem 110 via a physical host interface. Examples of the physical host interface include, but are not limited to, a Serial Advanced Technology Attachment (SATA) interface, a Peripheral Component Interconnect Express (PCIe) interface, a Universal Serial Bus (USB) interface, a Fibre Channel, a Serial Attached SCSI (SAS), a Double Data Rate (DDR) memory bus, a Small Computer System Interface (SCSI), a Dual In-line Memory Module (DIMM) interface (e.g., a DIMM socket interface supporting Double Data Rate (DDR)), etc. The physical host interface may be used to transfer data between the host system 120 and the memory subsystem 110. When the memory subsystem 110 is coupled to the host system 120 through a physical host interface (e.g., a PCIe bus), the host system 120 may also utilize an NVM Express (NVMe) interface to access components (e.g., the memory device 130). The physical host interface may provide an interface for transferring control, address, data, and other signals between the memory subsystem 110 and the host system 120. Figure 1 Memory subsystem 110 is illustrated as an example. In general, host system 120 can access multiple memory subsystems via the same communication connection, multiple separate communication connections, and / or a combination of communication connections.

[0027] The memory devices 130, 140 may include any combination of different types of non-volatile memory devices and / or volatile memory devices. Volatile memory devices (e.g., memory device 140) may be, but are not limited to, random access memory (RAM), such as dynamic random access memory (DRAM) and synchronous dynamic random access memory (SDRAM).

[0028] Some examples of non-volatile memory devices (e.g., memory device 130) include NAND-type flash memory and write-in-place memory, such as a three-dimensional cross-point ("3D cross-point") memory device, which is a cross-point array of non-volatile memory cells. The cross-point array of non-volatile memory can be combined with a stackable cross-grid data access array to perform bit storage based on changes in body resistance. In addition, in contrast to many flash-based memories, cross-point non-volatile memory can perform write-in-place operations, where non-volatile memory cells can be programmed without pre-erasing the non-volatile memory cells. NAND-type flash memory includes, for example, two-dimensional NAND (2D NAND) and three-dimensional NAND (3D NAND).

[0029] Each of the memory devices 130 may include one or more memory cell arrays. One type of memory cell, for example, a single level cell (SLC) may store one bit per cell. Other types of memory cells, such as multi-level cells (MLC), three-level cells (TLC), four-level cells (QLC), and five-level cells (PLC) may store multiple bits per cell. In some embodiments, each of the memory devices 130 may include one or more memory cell arrays, such as SLC, MLC, TLC, QLC, or any combination of such memory cell arrays. In some embodiments, a particular memory device may include an SLC portion of a memory cell, and an MLC portion, a TLC portion, a QLC portion, or a PLC portion. The memory cells of the memory device 130 may be grouped into pages, which may refer to logical units of a memory device for storing data. For some types of memory (e.g., NAND), pages may be grouped to form blocks.

[0030] Although nonvolatile memory components such as a 3D cross-point nonvolatile memory cell array and NAND-type flash memory (e.g., 2D NAND, 3D NAND) are described, the memory device 130 may be based on any other type of nonvolatile memory, such as read-only memory (ROM), phase-change memory (PCM), self-selected memory, other chalcogenide-based memories, ferroelectric transistor random access memory (FeTRAM), ferroelectric random access memory (FeRAM), magnetic random access memory (MRAM), spin transfer torque (STT)-MRAM, conductive bridging RAM (CBRAM), resistive random access memory (RRAM), oxide-based RRAM (OxRAM), or non-(NOR) flash memory, and electrically erasable programmable read-only memory (EEPROM).

[0031] The memory subsystem controller 115 (for simplicity, controller 115) can communicate with the memory device 130 to perform operations, such as reading data, writing data, or erasing data and other such operations at the memory device 130. The memory subsystem controller 115 can include hardware, such as one or more integrated circuits and / or discrete components, buffer memory, or a combination thereof. The hardware can include digital circuits with dedicated (i.e., hard-coded) logic to perform the operations described herein. The memory subsystem controller 115 can be a microcontroller, a dedicated logic circuit (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.), or other suitable processor.

[0032] The memory subsystem controller 115 may be a processing device that includes one or more processors (e.g., processor 117) configured to execute instructions stored in a local memory 119. In the illustrated example, the local memory 119 of the memory subsystem controller 115 includes an embedded memory configured to store instructions for performing various processes, operations, logic flows, and routines that control the operation of the memory subsystem 110, including handling communications between the memory subsystem 110 and the host system 120.

[0033] In some embodiments, local memory 119 may include memory registers that store memory pointers, fetched data, etc. Local memory 119 may also include read-only memory (ROM) for storing microcode. Figure 1 The example memory subsystem 110 in FIG. 1 has been illustrated as including a memory subsystem controller 115, but in another embodiment of the present disclosure, the memory subsystem 110 does not include a memory subsystem controller 115, but may rely on external control (e.g., provided by an external host or by a processor or controller separate from the memory subsystem).

[0034] Typically, the memory subsystem controller 115 may receive commands or operations from the host system 120, and may convert the commands or operations into instructions or appropriate commands to achieve the desired access to the memory device 130. The memory subsystem controller 115 may be responsible for other operations, such as wear leveling operations, garbage collection operations, error detection and error correction code (ECC) operations, encryption operations, cache operations, and address translation between logical addresses (e.g., logical block addresses (LBA), name space) and physical addresses (e.g., physical MU addresses, physical block addresses) associated with the memory device 130. The memory subsystem controller 115 may further include a host interface circuit system to communicate with the host system 120 via a physical host interface. The host interface circuit system may convert commands received from the host system into command instructions to access the memory device 130, and convert responses associated with the memory device 130 into information for the host system 120.

[0035] The memory subsystem 110 may also include additional circuits or components not illustrated. In some embodiments, the memory subsystem 110 may include a cache or buffer (e.g., DRAM) and address circuitry (e.g., row decoders and column decoders) that may receive addresses from the memory subsystem controller 115 and decode the addresses to access the memory device 130.

[0036] In some embodiments, the memory device 130 includes a local media controller 135 that operates in conjunction with the memory subsystem controller 115 to perform operations on one or more memory cells of the memory device 130. An external controller (e.g., the memory subsystem controller 115) may externally manage the memory device 130 (e.g., perform media management operations on the memory device 130). In some embodiments, the memory device 130 is a managed memory device, which is a raw memory device combined with a local controller (e.g., the local controller 135) to perform media management within the same memory device package. An example of a managed memory device is a managed NAND (MNAND) device.

[0037] The memory subsystem controller 115 may additionally include a QoS management component 113 that may be used to facilitate quality of service control for the logical devices of the memory subsystem 110. In some embodiments, the memory subsystem controller 115 includes at least a portion of the QoS management component 113. For example, the memory subsystem controller 115 may include a processor 117 (e.g., a processing device) that is configured to execute instructions stored in a local memory 119 for performing the operations described herein. In some embodiments, the QoS management component 113 is part of the host system 120, an application, or an operating system. In other embodiments, the local media controller 135 includes at least a portion of the QoS management component 113 and is configured to perform the functionality described herein.

[0038] The QoS management component 113 may receive bandwidth and IOPS parameters associated with logical device partitions for physical devices (e.g., memory devices 130, 140 of the memory subsystem 110). The QoS management component 113 may divide the bandwidth and / or IOPS capabilities for the memory devices 130, 140 into sub-bands, where each sub-band is associated with one of the logical devices. The QoS manager may then determine the I / O operations in the queue for a particular logical device and perform the queued operations using an earned credit scheme, where the logical device earns credits based on its associated QoS parameters. In various embodiments, the QoS management component 113 may operate with awareness of other logical devices associated with the memory devices 130, 140, so that any underutilized bandwidth or IOPS capabilities of one logical device can be redirected to another logical device during periods of more active workloads. Additional details regarding the operation of the QoS management component 113 are described below.

[0039] Figure 2An example interface between a host system 120 and a memory subsystem controller 115 implementing a QoS management component 113 is illustrated according to some embodiments of the present disclosure. In various implementations, the host system 120, the memory subsystem controller 115, and the QoS management component 113 correspond to Figure 1 The host system 120, the memory subsystem controller 115 and the QoS management component 113.

[0040] In some implementations, the memory subsystem controller 115 includes an NVMe controller 211 coupled to a PCIe port 210 that enables communication with the host system 120 across the PCIe bus 205. As described above, the memory subsystem may include a memory device that is partitioned into a plurality of logical devices 230 (e.g., logical devices 230-A, 230-B, 230-C). As described above, in various implementations, the resources (e.g., storage capacity, I / O bandwidth capacity, etc.) of a physical memory device such as a memory device 130 may be partitioned into a plurality of logical devices 230, wherein each logical device 230 represents a portion of the resources of the physical device. Each logical device 230 may be presented to the host system 120 as a representation of an independent physical device. In such examples, the total capacity of the physical device may be apportioned to the logical devices 230 (sometimes referred to as virtualized "physical functions"), either evenly across the logical devices 230 or with different capacity profiles or configurations based on a particular client or customer implementation. For example, a 1 terabyte (Tb) physical device may be partitioned into multiple logical devices of equivalent capacity (e.g., 10 partitions of 100 gigabytes (Gb), 1000 partitions of 1 Gb, etc.) or different capacities (e.g., 5 partitions of 100 Gb and 20 partitions of 25 Gb). Similarly, the bandwidth and / or IOPS capacity of a memory device may be divided across logical partitions 230. In one embodiment, bandwidth represents the total processing capacity of a memory device based on the amount of data in a given time period, while IOPS capacity represents the number of individual transactions that can be executed in a given time period, and is potentially affected by the size of those transactions. For example, a memory device with a bandwidth capacity of 6 Gb / sec may be partitioned such that logical device 230-A is configured with a maximum bandwidth of 2 Gb / sec, while logical device 230-B is configured with 1 Gb / sec. In other embodiments, other configurations may be used for logical devices 230.

[0041] In some implementations, the NVMe controller 211 may be configured with one or more logical or virtual NVMe controllers each associated with one of the logical devices 230. Additionally, each logical device 230 may be associated with a corresponding first-in, first-out (FIFO) command queue (e.g., FIFOs 212-A, 212-B, 212-C) that queues I / O commands for the logical device 230. As I / O commands are received from the host 120 via the PCIe port 210, the NVMe controller 211 may determine the destination logical device 230 to which the command will be directed and add the command to the corresponding FIFO queue for the destination logical device 230.

[0042] In various implementations, an I / O command received from the host 120 may be directed to one or more data blocks stored by the logical device 230. For example, a single I / O command (e.g., a command directed to a memory device configured as an NVMe device) may access a single data block stored on the logical device 230-A. In such an example, the NVMe controller 211 may determine that the command is directed to the logical device 230-A, and may store the command in a FIFO queue (e.g., FIFO212-A) for the logical device 230-A. Once the command is processed by the memory subsystem controller 115, the single NVMe I / O command may be translated into a format understood by the media controller of the memory device (e.g., a front-end media controller (FEMC) command). In some implementations, a single NVMe I / O command may be a command to access data consisting of multiple blocks. In such a case, one NVMe I / O command may later be translated into multiple FEMC commands processed by the memory device. For example, a single NVMe I / O command for 4 blocks of data may cause the memory device to later execute 4 FEMC commands.

[0043] In various implementations, each of the logical devices 230 may be configured with individual quality of service (QoS) parameters. As described above, the QoS parameters for the logical devices may be based on the bandwidth or IOPS capabilities of the memory devices that have been partitioned. For example, assuming that the memory devices are capable of providing a maximum bandwidth of 12 Gb / sec, the individual logical devices 230 may be partitioned and configured with a portion of that total capacity (e.g., 2 Gb / sec, 5 Gb / sec, etc.). In such examples, the total device bandwidth may be divided into sub-bands, with each logical device 230 being associated with one of the sub-bands. In some implementations, the entire bandwidth of the memory devices need not be apportioned across all logical devices. In other words, once a logical device has apportioned some of the bandwidth of the memory device, if additional bandwidth remains unallocated, it may remain reserved for use during periods where an increase in I / O workload is observed.

[0044] The logical device 230 may be configured with QoS parameters associated with a particular operating mode. Depending on the embodiment and mode, separate thresholds may be configured for different parameters (i.e., for bandwidth and IOPS). In one example, the logical device may be configured to not exceed a specified bandwidth or IOPS value (e.g., "limited mode"). The restricted mode sets the maximum performance cap (e.g., limit threshold) for the logical device that the QoS management component 113 may impose on each physical function. In some embodiments, each assigned limit should be set to less than the maximum bandwidth of the device (e.g., no logical device may have a performance cap that is higher than the drive capability). However, in some embodiments, the sum of all limits for all logical devices may exceed the maximum capacity of the memory device (referred to as "out of supply"). In one embodiment, there may be separate limit thresholds corresponding to bandwidth and IOPS. In such embodiments, the IOPS limit threshold should not exceed the bandwidth limit threshold, and the QoS management component 113 may impose both thresholds.

[0045] In another example, the logical device may be configured to not drop below a specified bandwidth or IOPS value (e.g., "reservation mode"). The reservation mode sets a minimum performance floor (e.g., a reservation threshold) for the logical device that the QoS management component 113 will guarantee for each physical function (assuming the host workload intensity allows the threshold to be met). In some implementations, the sum of all reservations should not exceed the maximum capacity of the memory device (referred to as "overprovisioning"). In one embodiment, there may be separate reservation thresholds for bandwidth and IOPS. In such embodiments, the IOPS reservation threshold should not be higher than the bandwidth reservation threshold. Depending on the received workload, the QoS management component 113 may guarantee at least one of these thresholds.

[0046] In another example, a logical device may be configured with a combination of limits and reservations (e.g., "reservation mode with limits"). This mode defines the performance window within which the logical device should execute, thereby defining the lower and upper bandwidth and IOPs values ​​that the QoS management component 113 can impose on each physical function. As described above, the corresponding IOPS threshold should not be higher than the corresponding bandwidth threshold.

[0047] The QoS management component 113 is responsible for managing QoS parameters for each of the logical devices 230 of the memory subsystem. The QoS management component determines whether there are any queued commands in the FIFO 212, and if so, performs one or more I / O operations for the logical device while complying with the QoS mode for the associated logical device 230. The QoS management component 113 facilitates this process by implementing a credit scheme, where credits are earned for each logical device according to its assigned QoS performance mode as described above. A timer (e.g., a clock) is configured for each logical device so that the logical device earns credits based on the timer cycle based on a reservation threshold for the logical device. Thus, the logical device may be configured to earn credits at a rate associated with the QoS parameters for the logical device (e.g., at a rate associated with the bandwidth frequency for the QoS mode) based on the cycle of the timer. In one embodiment, this timer may be referred to as a variable frequency credit replenishment clock.

[0048] The QoS management component 113 may include a QoS credit manager 225 to manage credits for each of the logical devices 230. In some embodiments, credits may be used to process a portion of the data associated with an I / O command. As described above, the logical device associated with the NVMe memory device may be configured so that each I / O command can access data consisting of one or more blocks. In such an example, the credit scheme may be configured so that the cost of accessing one block is one credit. Therefore, each block of data associated with the I / O operation requires one credit in order to complete the entire I / O operation. For example, if the I / O operation is associated with 10 blocks, 10 credits may be used to complete the I / O operation. The QoS credit manager 225 may identify QoS parameters for the logical device 230 and configure its associated timer so that the logical device can win credits based on the associated reserved performance bandwidth or IOPs setting (based on the data size of the I / O command). The QoS credit manager 225 may additionally deduct credits spent by the received I / O operation for a particular logical device 230. In some implementations, the QoS credit manager 225 may be configured to initiate each logical device 230 with a starting credit value greater than zero. Configuring the credit scheme in this manner may facilitate initial command processing more efficiently because the logical devices do not need to wait for an extended period of time to accumulate enough credits to perform an I / O operation. Figure 3 An example of credit management for a logical device is described.

[0049] In various embodiments, credits earned by one logical device 230 may be shared with another logical device 230. In other words, if logical device 230-A experiences a period of heavy workload that results in exhausting its credit balance (e.g., no available credits for queued I / O commands), it may utilize storage credits accumulated by logical devices 230-B or 230-C, provided those logical devices do not have any queued I / O commands in their associated FIFOs 212. Similarly, if there is any unallocated bandwidth for a memory device, it may also be utilized by logical device 230-A. In this latter case, the unallocated bandwidth may be assigned a logical device timer to maintain a credit store that may be shared with the allocated logical devices during periods of intense workload.

[0050] The QoS management component 113 may further include an arbitrator component 220 to monitor the FIFOs 212 for received I / O commands and select the FIFO from which the next I / O command to be processed will be retrieved. In various embodiments, the arbitrator component 220 may select the FIFO 212 in a round-robin manner and utilize the credit information managed by the QoS credit manager 225 to determine whether to process the command from the selected FIFO 212. In one example, the arbitrator component 220 may select FIFO 212-A and determine whether FIFO 212-A has a command that can be obtained. If so, the arbitrator component 220 may access the credit information maintained by the QoS credit manager 225 to determine whether the logical device 230-A has at least one credit available. If so, the arbitrator component 220 may obtain the command and forward it to the logical device 230-A. If the command is for multiple blocks, the arbitrator component 220 may determine how many credits are needed to process all blocks associated with the command. If the number of credits required to process an I / O command exceeds the number of credits available to the logic device, all blocks associated with the command may be processed, but the total credits available may be driven negative. Negative credit values ​​may still be refreshed based on the timer clock.

[0051] If the arbiter component 220 determines that the logic device 230-A does not have any available credits to process the I / O command, several alternatives may be performed. In one example, the arbiter component 220 may skip the command and move to the next available FIFO 212 (e.g., FIFO 212-B) to repeat the process for that FIFO 212. Alternatively, the arbiter component 220 may determine whether any of the other logic devices 230 have credits available but do not have any commands queued in their associated FIFO 212 to be processed. In such an example, the arbiter component 220 may utilize the processing time slots for the idle logic devices to process the commands for the logic device 230-A. For example, the arbiter component 220 may determine that the logic device 230-B has credits available but does not have commands queued in FIFO 212-B, and therefore may still process the commands queued in FIFO 212-A for the logic device 230-A, even though the logic device 230-A has not accumulated enough credits to execute the command.

[0052] In various embodiments, the arbiter component 220 may be configured to perform arbitration on several different types of requests issued by or for a physical function (e.g., a logic device 230-A, 230-B, 230-C). In one example, the arbiter component 220 may perform arbitration based on a timer (e.g., "timer arbitration"). In such an example, the arbiter component 220 may select a FIFO 212 for the logic device 230 based on the credit timer frequency associated with the logic device and issue a request called a "timer request". Thus, a timer request may be generated that is synchronized with each cycle of the credit replenishment clock mentioned above. In other words, the logic device 230 needs to have at least one credit for selecting its associated FIFO. Otherwise, the FIFO for the logic device 230 is bypassed and the next FIFO is selected. When a timer request is serviced by the arbiter component 220, the number of timer requests is decremented by the number of acquired blocks and may become negative. Timer credits are replenished on each cycle of the credit replenishment clock.

[0053] In another example, the arbitrator component 220 may perform arbitration based on available opportunities to execute commands (e.g., "opportunistic arbitration"). In such an example, if the selected FIFO 212 does not contain a command, the arbitrator component 220 may select a different FIFO, such as FIFO 212-A, that does have the command queued for processing regardless of whether the associated logical device has sufficient credits. This action may be referred to as an "opportunistic request" and is generated when a physical function has at least one IOPS excess credit and one bandwidth excess credit available. However, the arbitrator component 220 will only service an opportunity request when a timer request for another physical function has not yet been consumed, because there are no I / O operations for the physical function at that time.

[0054] In another example, the arbitrator component 220 may perform arbitration based on request bursts (e.g., "burst arbitration"). In such an example, if commands are acquired based on a timer (e.g., timer arbitration), then additional commands may be acquired immediately and processed. This action may be referred to as a "burst request" and is generated asynchronously between timer events when at least one burst credit (or timer credit) is available. Similar to a timer request, when a burst request is serviced by the arbitrator component 220, the number of burst credits is decremented by the number of acquired blocks and may become negative. In one embodiment, burst requests for all physical functions are serviced in a round-robin manner. Burst arbitration allows each physical function to process short I / O bursts faster and may be beneficial depending on the host workload. In a continuous type of workload, the burst request is consumed immediately, and then for each burst credit returned, only the timer request is processed.

[0055] As described above, opportunity requests utilize timer requests that are not yet consumed by other physical functions. When the sum of all reservations is strictly in oversupply, the device bandwidth that is still not assigned can be used by the QoS management component 113 to generate timer events that can be shared by all physical functions for opportunity requests. These opportunity requests can be issued by physical functions that have been configured in reservation mode or reservation mode with restrictions and have additional credit margins to allow them to reach the corresponding restrictions. In order to allow the IOPS limit to be reached, for example, the IOPS additional credit margin can be calculated as the difference between the IOPS limit and the IOPS reservation. Similarly, the bandwidth additional credit margin is calculated as the additional number of blocks that need to be transmitted above the bandwidth reservation so that the bandwidth limit can be reached.

[0056] In one embodiment, the bandwidth and IOPS extra credits are calculated for use during a certain time period, which may be referred to as a "session". One session follows another, and the extra credits are replenished at the beginning of each session. During a session, a physical function makes an opportunistic request as long as it has at least one available credit in each type of extra credit. When an opportunistic request is serviced, the IOPS extra credit is decremented by one, and the bandwidth extra credit is decremented by the number of blocks requested.

[0057] Figure 3 An example of credit management for a logical device according to some embodiments of the present disclosure is described. In some embodiments, this process may be performed by Figure 1 and as above relative to Figure 2 The QoS management component 113 described performs.

[0058] like Figure 3 As shown, the logical device has been configured such that its number of credits has been initialized to 10 at startup (e.g., timer cycle 300). The logical device is associated with a credit timer 330, wherein credits are earned for the logical device according to the QoS parameters associated with the logical device. Thus, as shown, after each successive timer cycle, additional credits are earned for the logical device, which additional credits can be applied to process I / O requests held in the corresponding FIFO queue. Figure 3 A series of I / O commands retrieved from an associated FIFO, the credit costs associated with processing the commands, and the credit accumulation resulting from the credit timer cycles of the credit timer 330 are illustrated.

[0059] In the timer cycle 301, the QoS management component (e.g., Figure 2 An arbiter component of the arbiter component 220 in the arbiter component 220 selects a FIFO for a logical device and identifies an I / O command 320 in the corresponding FIFO queue. In one embodiment, the I / O command 320 includes 8 sub-commands for different blocks. Alternatively, the I / O command 320 may represent a single sub-command that accesses 8 different blocks. In both cases, since the I / O command 320 accesses 8 blocks, a total of 8 credits will be required to satisfy the command. The first sub-command of the I / O command 320 is processed in the timer cycle 301, thereby decrementing the total credits from 10 to 9. As shown, 8 sub-commands for 8 blocks of the I / O command 320 are processed between clock cycles 301 and 302, thereby decrementing the credit count from 10 to 2 (shown by the "credit after subtraction" value of 2).

[0060] In cycle 302, the credit timer 330 has accumulated additional credits, which changes the credit count from 2 to 3 (illustrated by the "credits after increment" value of 3). In cycle 303, the credit timer 330 has accumulated additional credits, which changes the credit count from 3 to 4 (illustrated by the "credits after increment" value of 4). In some implementations, since there are no additional commands in the queue between timer cycles 302 and 303A, the arbiter component may select another FIFO queue to determine if there are any commands available for processing in the other FIFO queue.

[0061] In loop 303A, a second set of I / O commands 321 is obtained by the arbiter component 220. For example, the arbiter component 220 may obtain the second set of I / O commands 321 based on a burst arbitration scheme. As shown, the first subcommand decrements the credit count from 4 to 3 (illustrated by the "credits after subtraction" value of 3). In loop 304, additional credits are accumulated, which changes the credit count from 3 to 4 (illustrated by the "credits after increment" value of 4), and the credits are immediately used by the second subcommand of I / O command 321, which changes the credit count back to 3. The two additional subcommands are then processed, which further decrements the credit count to 1 (illustrated by the "credits after subtraction" value of 1) in loop 304A.

[0062] In loop 305, the credit timer 330 has accumulated additional credits, which changes the credit count from 1 to 2 (shown by the "credit after increment" value of 2). During the same loop, the arbiter component obtains a third set of I / O commands 322. The first subcommand is processed, which decrements the credit count from 2 to 1 (shown by the "credit after subtraction" value of 1). The second subcommand is then processed, which further decrements the credit count from 1 to 0 (shown by the "credit after subtraction" value of 0) in loop 305A.

[0063] In loop 305A, although the logic device has exhausted its earned credits, it can still process the remaining two subcommands of I / O command 322. Therefore, through loop 305B, the credit count is decremented from 0 to -2. Figure 3 As shown, one additional credit is accumulated in loop 306, and another additional credit is accumulated in loop 307, bringing the total credit count back to 0. Another credit is accumulated in loop 308, bringing the total credit count from 0 to 1 (shown by the "credit after increment" value of 1). As long as the logical device has one or more earned credits, the arbitrator component 220 can obtain new I / O commands associated with the logical device to be processed. By implementing a credit timer based on the QoS parameters for the logical device, an upper threshold for the logical device can be maintained.

[0064] Figure 4An example physical host interface between a host system and a memory subsystem implementing quality of service management according to some embodiments of the present disclosure is illustrated. In one embodiment, a memory subsystem controller 115 of a memory subsystem 110 is connected to a host system 120 via a physical host interface such as a PCIe bus 410. In one embodiment, the memory subsystem controller 115 manages one or more NVMe controllers 402-408. Depending on the embodiment, there may be a single NVMe controller, as described above with respect to Figure 2 As illustrated, a plurality of virtual NVMe controllers 402-408, which are virtual entities that appear as physical controllers to other devices such as the host system 120, are connected to the PCIe bus 410 by way of physical functions 412-418 associated with each virtual NVMe controller 402-408. Figure 4 Four virtual NVMe controllers 402-408 and four corresponding physical functions 412-418 are illustrated. However, in other embodiments, there may be any other number of NVMe controllers, each with a corresponding physical function. All virtual NVMe controllers 402-408 have the same priority and the same functionality.

[0065] Each of the virtual NVMe controllers 402-408 manages storage access operations for the underlying memory device 130. For example, the virtual NVMe controller 402 may receive a data access request from the host system 120 via the PCIe bus 410, including a request to read, write, or erase data. In response to the request, the virtual NVMe controller 402 may identify a physical memory address in the memory device 130 that is related to a logical memory address in the request, perform the requested memory access operation on the data stored at the physical address, and return the requested data and / or confirmation or error message to the host system 120 when appropriate. The virtual NVMe controllers 404-408 may function in the same or similar manner with respect to data access requests to the memory device 130.

[0066] As described above, each of the physical functions 412-418 is associated with each of the virtual NVMe controllers 402-408 so as to allow each virtual NVMe controller 402-408 to appear as a physical controller on the PCIe bus 410. For example, the physical function 412 may correspond to the virtual NVMe controller 402, the physical function 414 may correspond to the virtual NVMe controller 404, the physical function 416 may correspond to the virtual NVMe controller 406, and the physical function 418 may correspond to the virtual NVMe controller 408. The physical functions 412-418 are fully featured PCIe functions that can be discovered, managed, and manipulated similar to any other PCIe device, and thus can be used to configure and control PCIe devices (e.g., virtual NVMe controllers 402-408). Each physical function 412-418 may have a certain number of virtual functions associated with it. A virtual function is a lightweight PCIe function that shares one or more resources with a physical function and a virtual function associated with the physical function. Each virtual function has a PCI memory space that is used to map its register set. A virtual function device driver operates on the register set to implement its functionality, and the virtual function appears as an actual PCIe device accessible by the host system 120 through the PCIe bus 410.

[0067] Each physical function 412-418 may be assigned to any of the virtual machines 432-436 in the host system 120. When receiving I / O data from the virtual machines 432-436 at the virtual NVMe controllers 402-408, the virtual machine driver provides a guest physical address for the corresponding read / write command. The physical function number may be translated into a bus, device, and function (BDF) number, and then added to a direct memory access (DMA) operation to perform a DMA operation on the guest physical address. In one embodiment, the controller 115 further transforms the guest physical address into a system physical address for the memory subsystem 110.

[0068] In addition, each physical function 412-418 can be implemented in privileged mode or normal mode. When implemented in privileged mode, the physical function has a single management point that can control resource manipulation and storage configuration for other functions implemented in normal mode. In addition, the physical function in privileged mode can perform management options, including, for example, enabling / disabling multiple physical functions, storage configuration and quality of service (QoS) configuration, firmware and controller updates, vendor-specific statistics and events, diagnostics, secure erase / encryption, etc. Typically, the first physical function can implement privileged mode, and the rest of the physical functions can implement normal mode. However, in other embodiments, any of the physical functions can be configured to operate in privileged mode. Therefore, there can be one or more functions running in privileged mode.

[0069] The host system 120 runs multiple virtual machines 432, 434, 436 by executing a software layer 424, often referred to as a "hypervisor," above the hardware and below the virtual machines, such as Figure 4 Schematically shown in . In one illustrative example, the hypervisor 424 may be a component of a host operating system 422 executed by the host system 120. Alternatively, the hypervisor 424 may be provided by an application running under the host operating system 422, or may run directly on the host system 120 without an operating system underneath it. The hypervisor 424 may abstract the physical layer, including processors, memory, and I / O devices, and present this abstraction as a virtual device to the virtual machines 432, 434, 436, which include virtual processors, virtual memory, and virtual I / O devices. The virtual machines 432, 434, 436 may each execute a guest operating system that may utilize the underlying virtual devices, which may, for example, be mapped to a memory device 130 managed by one of the virtual NVMe controllers 402-408 in the memory subsystem 110. One or more applications may run on each virtual machine under the guest operating system.

[0070] Each virtual machine 432, 434, 436 may include one or more virtual processors. Processor virtualization may be implemented by scheduling time slots on one or more physical processors through the hypervisor 424 so that from the perspective of the guest operating system, those time slots are scheduled on virtual processors. Memory virtualization may be implemented through page tables (PTs), which are memory structures that translate guest memory addresses into physical memory addresses. The hypervisor 424 may run at a higher privilege level than the guest operating system, and the guest operating system may run at a higher privilege level than the guest application.

[0071] In one embodiment, there may be multiple partitions on the host system 120 representing virtual machines 432, 434, 436. The parent partition corresponding to virtual machine 432 is a root partition (i.e., root ring 0), which has additional privileges to control the life cycle of other child partitions (i.e., conventional ring 0) corresponding to, for example, virtual machines 434 and 436. Each partition has a corresponding virtual memory, and instead of presenting a virtual device, the child partition sees the physical device assigned to it. When the host system 120 is initially started, the parent partition can directly see all physical devices. A pass-through mechanism (e.g., PCIe pass-through or direct device assignment) allows the parent partition to assign an NVMe device (e.g., one of the virtual NVMe controllers 402-408) to a child partition. The associated virtual NVMe controllers 402-408 can appear as virtual storage resources to each of the virtual machines 432, 434, 436, and the guest operating system or the guest application running therein can access the virtual storage resources. In one embodiment, for example, virtual machine 432 is associated with virtual NVMe controller 402, virtual machine 434 is associated with virtual NVMe controller 404, and virtual machine 436 is associated with virtual NVMe controller 406. In other embodiments, one virtual machine may be associated with two or more virtual NVMe controllers. Virtual machines 432, 434, 436 may identify associated virtual NVMe controllers using corresponding bus, device, and function (BDF) numbers, as will be described in more detail below.

[0072] The QoS management component 113 may implement access control services for each of the virtual NVMe controllers 402-408. The access control service manages devices that have access permissions for the virtual NVMe controllers 402-408. The access permissions may define, for example, which of the virtual machines 432-436 on the host system 120 can access each of the virtual NVMe controllers 402-408, and what operations each of the virtual machines 432-436 can perform on each of the virtual NVMe controllers 402-408. In one embodiment, the QoS management component 113 controls access permissions for each of the virtual NVMe controllers 402-408 individually. For example, in privileged mode, the QoS management component 113 may grant permission to the virtual machine 432 to read and write data using the virtual NVMe controller 402, and permission to only read data using the virtual NVMe controller 404. Similarly, in privileged mode, the QoS management component 113 may grant permission to the virtual machine 432 to read and write data using only the virtual NVMe controller 404. Any combination of access permissions may be defined for the virtual NVMe controllers 402-408. When a memory access request is received for one of the virtual NVMe controllers 402-408, the QoS management component 113 may analyze the conditions of the request (e.g., requester, target, operation, requested data address, etc.) based on the access policy that defines the access control service. The access policy may be stored in the local memory 119. If the request satisfies the corresponding access policy (the conditions of the request match the conditions specified in the corresponding access policy), the QoS management component 113 may grant the access request. Otherwise, the request may be denied.

[0073] Figure 5 A method for quality of service control of a logic device of a memory subsystem according to some embodiments of the present disclosure is described. The method 500 may be performed by a processing logic that may include hardware (e.g., a processing device, a circuit system, a dedicated logic, a programmable logic, a microcode, a hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executed on a processing device), or a combination thereof. In some embodiments, the method 500 is performed by Figure 1 113. Although shown in a particular sequence or order, the order of the processes may be modified unless otherwise specified. Therefore, it should be understood that the illustrated embodiments are merely examples, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. In addition, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in each embodiment. Other process flows are also possible.

[0074] At operation 510, processing logic provides several physical functions 412-418, where each of the physical functions 212-218 corresponds to a virtual memory controller, such as NVMe controller 211 or one of virtual NVMe controllers 402-408. Each of the physical functions 412-418 represents a corresponding logical device, such as one of logical devices 230-A to 230-C, to the host system 120 as a physical device over a communication interface, such as PCIe bus 210. In one embodiment, the physical functions 412-418 are created in response to input received from a system administrator via a management interface.

[0075] At operation 520, processing logic presents physical functions 412-418 to a host computing system, such as host system 120, via a communication interface, such as PCIe bus 210. In one embodiment, host system 120 assigns each of physical functions 412-418 to a different virtual machine, such as one of virtual machines 432, 434, 436, running on host system 120. Each of physical functions 412-418 provides a configuration space for a corresponding one of logical devices, where each configuration space is individually addressable. In other embodiments, each of physical functions 412-418 may be assigned to a separate application, user, program, etc., running on host system 120, or even entirely to a separate host system.

[0076] At operation 530, processing logic receives a request to perform an I / O operation from the host system 120. In one embodiment, the request is received from a designated virtual machine running on the host system 120, such as one of the virtual machines 432, 434, 436. The request may involve one or more memory devices, such as the memory device 130.

[0077] At operation 540, the processing logic iteratively processes I / O operations corresponding to the logical devices 230-A to 230-C associated with the memory device 130. In one embodiment, one or more of the virtual NVMe controllers 402-408 may perform the requested I / O operation, such as a read, write, or erase operation, and may transmit the requested data and / or confirmation or error messages back to the host system 120 when appropriate. In one embodiment, the arbiter component 220 of the QoS management component 220 manages memory access requests and background operations performed on the memory devices in view of QoS parameters associated with the physical functions. In one embodiment, the QoS credit manager 225 manages operation credits for the physical functions, wherein the number of credits associated with each physical function is based at least in part on the QoS parameters, such as relative to the physical function. Figure 6For example, QoS management component 220 incrementally rotates through each of the logical devices to determine whether there are I / O operations for the current logical device (eg, in the corresponding FIFO) and whether sufficient operation credits are available to process those I / O operations.

[0078] Figure 6 A method of processing I / O operations for a logic device of a memory device in view of a quality of service parameter according to some embodiments of the present disclosure is described. The method 600 may be performed by a processing logic that may include hardware (e.g., a processing device, a circuit system, a dedicated logic, a programmable logic, a microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executed on a processing device), or a combination thereof. In some embodiments, the method 600 is performed by Figure 1 113. Although shown in a particular sequence or order, the order of the processes may be modified unless otherwise specified. Therefore, it should be understood that the illustrated embodiments are merely examples, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. In addition, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in each embodiment. Other process flows are also possible.

[0079] At operation 610, processing logic identifies a current logical device. In one embodiment, identifying the current logical device includes applying at least one of a burst arbitration scheme or a timer arbitration scheme to the logical device. The current logical device represents one of the logical devices 230-A through 230-C being operated by the QoS management component 113. As part of the iterative process, the QoS management component 113 operates on each of the logical devices in sequence, returns to the first logical device after operating on the last logical device, and the operations described with respect to method 600 may be performed for each of the logical devices.

[0080] At operation 620, processing logic determines one or more I / O operations in a queue for the current logical device. In one embodiment, the QoS management component 113 checks a corresponding queue for the current logical device, such as a corresponding FIFO 212-A to 212-C. The corresponding queue maintains requests to perform I / O operations at the current logical device. In one embodiment, the request is received from the host system 120 via the PCIe bus 205 and is directed to an associated physical function of the memory subsystem controller 115.

[0081] At operation 630, processing logic determines a number of operating credits associated with the current logical device. In one embodiment, the number of credits is based at least in part on a set of quality of service (QoS) parameters for the current logical device. In one embodiment, the set of QoS parameters for the current logical device indicates a subset of bandwidth capabilities or input / output operations per second (IOPS) capabilities of a memory device associated with the current logical device. In one embodiment, the set of QoS parameters for the current logical device includes at least one of a limit parameter associated with a maximum bandwidth and / or IOPS capability of the current logical device or a reservation parameter associated with a minimum bandwidth and / or IOPS capability of the current logical device.

[0082] At operation 640, processing logic determines whether the number of operating credits satisfies a threshold condition. In one embodiment, the number of operating credits satisfies the threshold condition when the number of operating credits is a positive number of operating credits. For example, the QoS management component 113 may determine that the threshold condition is satisfied as long as the current logical device has at least one operating credit. In other embodiments, some other threshold number of credits may be defined. In one embodiment, the number of operating credits associated with the current logical device is periodically increased in response to the elapse of one or more timer cycles of a timer associated with the current logical device, as described above with respect to Figure 3 Detailed description.

[0083] In response to determining that the number of operation credits satisfies the threshold condition, at operation 650, processing logic performs the one or more I / O operations for the current logic device. In one embodiment, each of the one or more I / O operations may include one or more sub-operations. Each sub-operation corresponds to a memory access operation, such as a read operation, a write / program operation, or an erase operation. In one embodiment, performing each of the sub-operations uses one operation credit and decrements the operation credit count accordingly.

[0084] At operation 660, processing logic identifies a subsequent logical device in the plurality of logical devices. Once the I / O operation is performed, or if it is determined at operation 640 that insufficient operation credits are available, the QoS management component 113 may identify a subsequent logical device to be operated on based on the associated arbitration scheme being implemented on the memory device 130.

[0085] Figure 7 An example machine illustrating a computer system 700 within which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed. In some embodiments, the computer system 700 may correspond to a computer system that includes, is coupled to, or utilizes a memory subsystem (e.g., Figure 1 A host system (eg, a memory subsystem 110) of Figure 1host system 120), or can be used to perform operations of the controller (for example, execute an operating system to perform operations corresponding to Figure 1 In some embodiments, the machine may be connected (e.g., using a network) to other machines in a LAN, an intranet, an extranet, and / or the Internet. The machine may operate in the capacity of a server or a client machine in a client-server network environment, or in the capacity of a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.

[0086] The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a network appliance, a server, a network router, a switch or bridge or a non-digital circuit system, or any machine capable of executing (sequentially or otherwise) a set of instructions that specify actions to be taken by the machine. In addition, while a single machine is described, the term "machine" should also be construed to include any collection of machines that individually or collectively execute one (or more) sets of instructions to perform any one or more of the methodologies discussed herein.

[0087] The example computer system 700 includes a processing device 702, a main memory 704 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM), such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), etc.), a static memory 706 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage system 718, which communicate with each other via a bus 730.

[0088] The processing device 702 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, and the like. More specifically, the processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. The processing device 702 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The processing device 702 is configured to execute instructions 726 for performing the operations and steps discussed herein. The computer system 700 may further include a network interface device 708 that communicates via a network 720.

[0089] The data storage system 718 may include a machine-readable storage medium 724 (also referred to as a computer-readable medium) on which is stored one or more sets of instructions 726 or software embodying any one or more of the methods or functions described herein. The instructions 726 may also reside, completely or at least partially, within the main memory 704 and / or within the processing device 702 during execution by the computer system 700, the main memory 704 and the processing device 702 also constituting machine-readable storage media. The machine-readable storage medium 724, the data storage system 718, and / or the main memory 704 may correspond to Figure 1 Memory subsystem 110.

[0090] In one embodiment, instructions 726 include instructions for implementing instructions corresponding to a parallel iterator component (e.g., Figure 1 113) of the QoS management component 113). Although the machine-readable storage medium 724 is shown as a single medium in the example embodiment, the term "machine-readable storage medium" should be considered to include a single medium or multiple media storing one or more sets of instructions. The term "machine-readable storage medium" should also be considered to include any medium capable of storing or encoding a set of instructions for execution by a machine and causing the machine to perform any one or more of the methods of the present disclosure. Therefore, the term "machine-readable storage medium" should be considered to include, but not limited to, solid-state memory, optical media, and magnetic media.

[0091] Some portions of the previous detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing can most effectively convey the substance of their work to others skilled in the art. In this article, and generally, an algorithm is conceived to be a self-consistent sequence of operations that produce a desired result. An operation is one that requires physical manipulation of physical quantities. Typically (but not necessarily), these quantities take the form of electrical or magnetic signals that can be stored, combined, compared, and otherwise manipulated. It has proven to be convenient at times, primarily for common reasons, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, and the like.

[0092] It should be borne in mind, however, that all of these and similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure may refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within a computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage systems.

[0093] The present disclosure also relates to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the desired purpose, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. This computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk, including a floppy disk, an optical disk, a CD-ROM and a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card or an optical card, or any type of medium suitable for storing electronic instructions and each coupled to a computer system bus.

[0094] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general purpose systems may be used in conjunction with the programs according to the teachings herein, or it may prove convenient to construct a more specialized device to perform the method. The structures of various such systems will be presented as set forth in the description below. In addition, the present disclosure is not described with reference to any particular programming language. It should be appreciated that the teachings of the present disclosure as described herein may be implemented using various programming languages.

[0095] The present disclosure may be provided as a computer program product or software, which may include a machine-readable medium having stored thereon instructions that can be used to program a computer system (or other electronic device) to perform a process according to the present disclosure. The machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). In some embodiments, the machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) readable storage medium, such as a read-only memory ("ROM"), a random access memory ("RAM"), a disk storage medium, an optical storage medium, a flash memory component, etc.

[0096] In the foregoing description, embodiments of the present disclosure have been described with reference to specific example embodiments thereof. It will be apparent that various modifications may be made to the present disclosure without departing from the broader spirit and scope of the embodiments of the present disclosure as set forth in the appended claims. Therefore, the description and drawings should be viewed in an illustrative sense rather than a restrictive sense.

Claims

1. A method comprising: receiving a plurality of requests to perform a plurality of input / output (I / O) operations corresponding to a plurality of logical devices associated with a memory device; assigning the plurality of requests to respective queues associated with the plurality of logical devices; as well as The plurality of requests are iteratively processed in view of respective numbers of operating credits associated with the plurality of logical devices, wherein the respective numbers of credits are based at least in part on respective sets of quality of service (QoS) parameters for the plurality of logical devices. 2 . The method of claim 1 , wherein iteratively processing the plurality of requests comprises identifying a current logical device among the plurality of logical devices by applying at least one of a burst arbitration scheme or a timer arbitration scheme to the plurality of logical devices.

3. The method of claim 1 , wherein the respective QoS parameter sets for the plurality of logical devices indicate respective subsets of at least one of a bandwidth capability or an input / output operations per second (IOPS) capability of the memory device associated with each logical device.

4. The method of claim 3 , wherein the respective QoS parameter sets for the plurality of logical devices comprise at least one of: a limit parameter associated with at least one of a maximum bandwidth capability or a maximum IOPS capability of each logical device, or a reservation parameter associated with at least one of a minimum bandwidth capability or a minimum IOPS capability of each logical device.

5. The method of claim 1 , wherein iteratively processing the plurality of requests comprises determining whether the respective number of operating credits satisfies a threshold condition, and wherein the respective number of operating credits satisfies the threshold condition when the number of operating credits is a positive number of operating credits.

6. The method of claim 1, wherein the respective number of operating credits is periodically increased in response to the elapse of one or more timer cycles of timers associated with the plurality of logic devices.

7. The method according to claim 1, further comprising: A plurality of physical functions are presented to a host computing system through a communication interface, wherein each of the plurality of physical functions represents a corresponding one of the plurality of logical devices to the host computing system as a physical device.

8. A system comprising: Memory device; as well as a processing device operatively coupled to the memory device, the processing device performing operations including: receiving a plurality of requests to perform a plurality of input / output (I / O) operations corresponding to a plurality of logical devices associated with the memory device; assigning the plurality of requests to respective queues associated with the plurality of logical devices; as well as The plurality of requests are iteratively processed in view of respective numbers of operating credits associated with the plurality of logical devices, wherein the respective numbers of credits are based at least in part on respective sets of quality of service (QoS) parameters for the plurality of logical devices.

9. The system of claim 8, wherein iteratively processing the plurality of requests comprises identifying a current logical device among the plurality of logical devices by applying at least one of a burst arbitration scheme or a timer arbitration scheme to the plurality of logical devices.

10. The system of claim 8, wherein the respective sets of QoS parameters for the plurality of logical devices indicate respective subsets of at least one of a bandwidth capability or an input / output operations per second (IOPS) capability of the memory device associated with each logical device.

11. The system of claim 10, wherein the respective QoS parameter sets for the plurality of logical devices comprise at least one of: a limit parameter associated with at least one of a maximum bandwidth capability or a maximum IOPS capability of each logical device, or a reservation parameter associated with at least one of a minimum bandwidth capability or a minimum IOPS capability of each logical device.

12. The system of claim 8, wherein iteratively processing the plurality of requests comprises determining whether the respective number of operating credits satisfies a threshold condition, and wherein the respective number of operating credits satisfies the threshold condition when the number of operating credits is a positive number of operating credits.

13. The system of claim 8, wherein the respective number of operating credits is periodically increased in response to the elapse of one or more timer cycles of timers associated with the plurality of logic devices.

14. The system of claim 8, wherein the operations performed by the processing device further comprise: A plurality of physical functions are presented to a host computing system through a communication interface, wherein each of the plurality of physical functions represents a corresponding one of the plurality of logical devices to the host computing system as a physical device.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising: receiving a plurality of requests to perform a plurality of input / output (I / O) operations corresponding to a plurality of logical devices associated with a memory device; assigning the plurality of requests to respective queues associated with the plurality of logical devices; as well as The plurality of requests are iteratively processed in view of respective numbers of operating credits associated with the plurality of logical devices, wherein the respective numbers of credits are based at least in part on respective sets of quality of service (QoS) parameters for the plurality of logical devices.

16. The non-transitory computer-readable storage medium of claim 15, wherein iteratively processing the plurality of requests comprises identifying a current logical device among the plurality of logical devices by applying at least one of a burst arbitration scheme or a timer arbitration scheme to the plurality of logical devices.

17. The non-transitory computer-readable storage medium of claim 15, wherein the respective QoS parameter sets for the plurality of logical devices indicate respective subsets of at least one of a bandwidth capability or an input / output operations per second (IOPS) capability of the memory device associated with each logical device.

18. The non-transitory computer-readable storage medium of claim 17, wherein the respective QoS parameter sets for the plurality of logical devices comprise at least one of: a limit parameter associated with at least one of a maximum bandwidth capability or a maximum IOPS capability of each logical device, or a reservation parameter associated with at least one of a minimum bandwidth capability or a minimum IOPS capability of each logical device.

19. The non-transitory computer-readable storage medium of claim 15, wherein iteratively processing the plurality of requests comprises determining whether the respective number of operating credits satisfies a threshold condition, and wherein the respective number of operating credits satisfies the threshold condition when the number of operating credits is a positive number of operating credits.

20. The non-transitory computer-readable storage medium of claim 15, wherein the respective number of operating credits is periodically increased in response to the elapse of one or more timer cycles of timers associated with the plurality of logic devices.

Images