Arbiter circuit in a memory subsystem for commands from multiple physical functions

Abstract

A system controller of a memory system may present a plurality of physical functions (PFs) to a host computing system. The system controller may store commands from the host in separate queues and issue the commands using an arbiter circuit. The arbiter may determine a difference between a quota of commands and a count of commands issued from a respective queue. The quota is derived from a share specified by the host for the respective PF. The arbiter circuit determines a subset of queues by excluding queues that are empty and queues with a negative difference. The arbiter circuit may randomly select a selected queue from the subset and issue a command from the selected queue.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to memory subsystems, and more particularly to arbiter circuits in a memory subsystem for commands from multiple physical functions (PFs). Background Art

[0002] The memory subsystem may be a storage system, such as a solid-state drive (SSD) or a hard disk drive (HDD). The memory subsystem may be a memory module, such as a dual in-line memory module (DIMM), a small outline DIMM (SO-DIMM), or a non-volatile dual in-line memory module (NVDIMM). The memory subsystem may include one or more memory components that store data. For example, the memory components may be non-volatile memory components and volatile memory components. In general, the host system may utilize the memory subsystem to store data at the memory components and retrieve data from the memory components. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The present disclosure will be more fully understood from the detailed description given below and the accompanying drawings of various embodiments of the present disclosure.

[0004] Figure 1 An example computing environment including a memory subsystem according to some embodiments of the present disclosure is described.

[0005] Figure 2 An example physical host interface between a host system and a memory subsystem implementing an arbiter component according to some embodiments of the present disclosure is illustrated.

[0006] Figure 3 Multiple queues, an arbiter component, and a single media controller in a memory subsystem according to some embodiments of the present disclosure are described.

[0007] Figure 4A Multiple queues, an arbiter component, and a single media controller in a memory subsystem according to some embodiments of the present disclosure are described.

[0008] Figure 4B A block diagram illustrating an arbiter circuit according to some embodiments of the present disclosure.

[0009] Figure 5 A sliding window of multiple sub-counters of a dynamic counter with an arbiter component in a memory subsystem is described according to some embodiments of the present disclosure.

[0010] Figure 6 A physical controller implementing an arbitrator component according to some embodiments of the present disclosure is described.

[0011] Figure 7is a flow chart of another example method for arbitrating commands from multiple queues associated with multiple physical functions in a memory subsystem according to some embodiments of the present disclosure.

[0012] Figure 8 is a block diagram of an example computer system in which embodiments of the present disclosure may operate. DETAILED DESCRIPTION

[0013] Aspects of the present disclosure relate to using an arbiter circuit for commands from multiple PFs in a memory subsystem. An example of a memory subsystem is a storage device coupled to a central processing unit (CPU) via a peripheral interconnect (e.g., an input / output bus, a storage area network). Examples of storage devices include solid-state drives (SSDs), flash drives, universal serial bus (USB) flash drives, and hard disk drives (HDDs). Another example of a memory subsystem is a memory module coupled to a CPU via a memory bus. Examples of memory modules include dual in-line memory modules (DIMMs), small outline DIMMs (SO-DIMMs), non-volatile dual in-line memory modules (NVDIMMs), etc. In some embodiments, the memory subsystem may be a hybrid memory / storage subsystem. In general, a host system may utilize a memory subsystem that includes one or more memory components. The host system may provide data to be stored at the memory subsystem and may request data to be retrieved from the memory subsystem.

[0014] The memory subsystem may include multiple memory components that can store data from the host system. In some systems, the memory subsystem presents itself to the host system as a single physical function through a peripheral component interconnect express (PCIe) interface. Some conventional memory subsystems utilize the single root input / output virtualization (SR-IOV) specification. SR-IOV is a specification that allows PCIe resources to be isolated among various hardware functions for manageability and performance reasons, while also allowing a single physical PCIe device to be shared in a virtual environment. SR-IOV provides different virtual functions for different virtual components (e.g., network adapters) on a physical server machine. SR-IOV also allows different virtual machines in a virtual environment to share a single PCIe hardware interface. The physical function allows several virtual functions to be enumerated, and the hypervisor can then assign the virtual function to one or more virtual machines. This solution requires two sets of drivers, including a physical function driver for enumerating virtual functions, and the kernel needs to support a complete stack with SR-IOV capabilities, and then the virtual function requires another driver that can only run virtual functions. These conventional memory subsystems do not present multiple physical functions to the host system through the PCIe interface.

[0015] Aspects of the present disclosure solve the above and other defects by having a memory subsystem that uses multiple physical functions, multiple corresponding queues, and an arbiter component for arbitrating commands from the multiple queues. Aspects of the present disclosure provide adaptability by allowing runtime workloads to be adjusted by updating arbitration parameters (e.g., percentage shares) based on statistics periodically. The host system can specify the number of physical functions to be used and the corresponding percentage shares for each of the physical functions of the number. Aspects of the present disclosure provide fairness in arbitration by servicing commands according to the percentage of shares specified by the host system. Aspects of the present disclosure provide opportunistic arbitration by fully utilizing bandwidth when some physical functions are idle (e.g., the corresponding queue has no commands). Aspects of the present disclosure provide decentralized arbitration by distributing the commands of the physical functions into timing windows in an attempt to prevent command clusters of a single physical function. In this way, the worst-case latency can be improved.

[0016] Advantages of the present disclosure include, but are not limited to, improving performance by utilizing the entire bandwidth between the host system and the memory subsystem and by distributing commands into timing windows, improving latency by allowing runtime workloads to adjust periodically based on statistics, and improving adaptability. In addition, by allowing the host system to specify corresponding shares of multiple physical functions, aspects of the present disclosure allow arbitration to be fair among multiple physical functions. In addition, by using an outstanding command queue and an arbitrator component to arbitrate the processing of commands in the outstanding command queue, the quality of service and performance of the memory subsystem can be improved. Figures 1 to 8 Provides additional details of the arbitrator component.

[0017] Figure 1 An example computing environment 100 including a memory subsystem 110 according to some embodiments of the present disclosure is illustrated. The memory subsystem 110 may include media such as memory components 112A-112N. The memory components 112A-112N may be volatile memory components, non-volatile memory components, or a combination thereof. In some embodiments, the memory subsystem is a storage system. An example of a storage system is an SSD. In some embodiments, the memory subsystem 110 is a hybrid memory / storage subsystem. In general, the computing environment 100 may include a host system 120 that uses the memory subsystem 110. For example, the host system 120 may write data to the memory subsystem 110 and read data from the memory subsystem 110.

[0018] The host system 120 may be a computing device such as a desktop computer, a laptop computer, a network server, a mobile device, or such computing devices including a memory and a processing device. The host system 120 may include or be coupled to the memory subsystem 110 so that the host system 120 can read data from the memory subsystem 110 or write data to the memory subsystem 110. The host system 120 may be coupled to the memory subsystem 110 via a physical host interface. As used herein, "coupled to" generally refers to a connection between components, which may be an indirect communication connection, wired or wireless, or a direct communication connection (e.g., without an intermediate component), including connections such as electrical, optical, magnetic, etc. Examples of physical host interfaces include, but are not limited to, a Serial Advanced Technology Attachment (SATA) interface, a Peripheral Component Interconnect Express (PCIe) interface, a Universal Serial Bus (USB) interface, a Fibre Channel, a Serial Attached SCSI (SAS), etc. The physical host interface may be used to transfer data between the host system 120 and the memory subsystem 110. The host system 120 may further utilize an NVM Express (NVMe) interface to access the memory components 112A-112N when the memory subsystem 110 is coupled to the host system 120 through a PCIe interface. The physical host interface may provide an interface for passing control, address, data, and other signals between the memory subsystem 110 and the host system 120.

[0019] Memory components 112A to 112N may include any combination of different types of nonvolatile memory components and / or volatile memory components. Examples of nonvolatile memory components include NAND-type flash memory. Each of memory components 112A to 112N may include one or more memory cell arrays, such as single-level cells (SLC) or multi-level cells (MLC) (e.g., three-level cells (TLC) or four-level cells (QLC)). In some embodiments, a particular memory component may include both an SLC portion and an MLC portion of a memory cell. Each of the memory cells may store one or more data bits (e.g., data blocks) used by the host system 120. Although nonvolatile memory components such as NAND-type flash memory are described, memory components 112A to 112N may be based on any other type of memory, such as volatile memory. In some embodiments, the memory components 112A to 112N may be, but are not limited to, random access memory (RAM), read-only memory (ROM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), phase change memory (PCM), magnetic random access memory (MRAM), or non-(NOR) flash memory, electrically erasable programmable read-only memory (EEPROM), and a cross-point array of non-volatile memory cells. The cross-point array of non-volatile memory may perform bit storage based on changes in body resistance in conjunction with a stackable cross-grid data access array. In addition, compared to many flash-based memories, the cross-point non-volatile memory may perform write-in-place operations, where non-volatile memory cells may be programmed without having to erase the non-volatile memory cells in advance. Furthermore, the memory cells of the memory components 112A to 112N may be grouped into memory pages or data blocks, which may refer to units of a memory component used to store data.

[0020] The memory system controller 115 (hereinafter referred to as the "controller") can communicate with the memory components 112A-112N to perform operations such as reading data, writing data, or erasing data at the memory components 112A-112N, as well as other such operations. The controller 115 may include hardware such as one or more integrated circuits and / or discrete components, buffer memory, or a combination thereof. The controller 115 may be a microcontroller, a dedicated logic circuit system (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.), or other suitable processor. The controller 115 may include a processor (processing device) 117 configured to execute instructions stored in a local memory 119. In the illustrated example, the local memory 119 of the controller 115 includes an embedded memory configured to store instructions for executing various processes, operations, logic flows, and routines that control the operation of the memory subsystem 110, including handling communications between the memory subsystem 110 and the host system 120. In some embodiments, the local memory 119 may include memory registers that store memory pointers, retrieved data, etc. Local memory 119 may also include read-only memory (ROM) for storing microcode. Figure 1 The example memory subsystem 110 in FIG. 1 has been described as including a controller 115, but in another embodiment of the present disclosure, the memory subsystem 110 may not include the controller 115, but may rely on external control (e.g., provided by an external host or by a processor or controller separate from the memory subsystem).

[0021] In general, the controller 115 may receive commands or operations from the host system 120, and may convert the commands or operations into instructions or appropriate commands to achieve the desired access to the memory components 112A to 112N. The controller 115 may be responsible for other operations such as wear leveling operations, obsolete item collection operations, error detection and error correction code (ECC) operations, encryption operations, cache operations, and address translation between logical block addresses and physical block addresses associated with the memory components 112A to 112N. The controller 115 may further include a host interface circuit system for communicating with the host system 120 via a physical host interface. The host interface circuit system may convert commands received from the host system into command instructions to access the memory components 112A to 112N, and convert responses associated with the memory components 112A to 112N into information for the host system 120.

[0022] The memory subsystem 110 may also include additional circuitry or components not illustrated. In some embodiments, the memory subsystem 110 may include a cache or buffer (e.g., DRAM) and address circuitry (e.g., row decoders and column decoders) that may receive addresses from the controller 115 and decode the addresses to access the memory components 112A-112N.

[0023] The memory subsystem 110 includes an arbitrator component 113 that can arbitrate commands received from the host system 120. In some embodiments, the controller 115 includes at least a portion of the arbitrator component 113. For example, the controller 115 may include a processor 117 (processing device) that is configured to execute instructions stored in a local memory 119 for performing the operations described herein. In some embodiments, some functionality of the arbitrator component 113 may be implemented as part of the processor 117, as part of firmware on the controller 115, as part of an application or operating system executed on the processor 117. Alternatively, some functionality of the arbitrator component 113 may be implemented as part of the host system 120.

[0024] The arbiter component 113 may use an outstanding command queue to store commands (e.g., read requests and write requests) to prevent data conflicts and enhance the quality of service for accessing data in the memory subsystem. The outstanding command queue may improve request traffic throughput based on different types of traffic in the memory subsystem. The controller 115 may use the arbiter component 113 to arbitrate which commands are selected from which queues for processing by the controller 115. The arbiter component 113 may include control logic and an outstanding command queue to provide ordered access to requested data and potentially unordered access to requested data. In some embodiments, the arbiter component 113 may receive preload instructions from one or more applications executing on the host system 120 to improve quality of service.

[0025] In some embodiments, to further improve performance and quality of service of the memory subsystem, the arbiter component 113 can determine when to perform scheduling of requests and fill operations based on when the requests are received and according to an arbitration scheme, as described herein.

[0026] In some embodiments, to further improve the performance and service quality of the memory subsystem, the arbiter component 113 may determine the difference between the quota of commands from one of the multiple queues processed by the controller 115 and the count of commands, wherein each queue is associated with one of the multiple PFs. The quota of commands for each of the multiple queues may be based on or derived from the share for the corresponding PF, which is specified by the host system 120. The arbiter component 113 may determine whether the corresponding queue has no commands. That is, the corresponding queue does not store uncompleted commands at the time of arbitration. The arbiter component 113 may use an empty indicator from the queue itself to determine that the queue is empty. Alternatively, the arbiter component 113 may use other techniques to determine the empty state of the queue. The arbiter component 113 may determine a subset of the multiple queues by excluding queues without commands and by excluding queues with negative numbers for the corresponding difference. That is, the subset does not include queues that are empty as determined above or queues with corresponding negative difference values. This results in full utilization of bandwidth by not selecting PFs that do not have any uncompleted queues (or are otherwise idle) and prioritizing queues that have not yet met their corresponding quotas. The arbiter component 113 may randomly select a selected queue from the subset and process commands from the selected queue, such as by issuing commands to a processor to process commands from the selected queue. Additional details of the arbiter component 113 are described below.

[0027] Figure 2An example physical host interface between a host system 120 and a memory subsystem implementing an arbiter component 113 according to some embodiments of the present disclosure is illustrated. In one embodiment, a controller 115 of the memory subsystem 110 is connected to the host system 120 via a physical host interface (e.g., a PCIe bus 210). In one embodiment, an arbiter component 113, which may be implemented as part of the controller 115 or as a circuit separate from the controller 115. The controller 115 manages a number of queues 202 to 208 and a number of physical functions (PFs) 212 to 218 within the controller 115. Each physical function 212 to 218 is a PCI function that supports the SR-IOV capability defined by the SR-IOV specification. The physical functions 212 to 218 are fully characterized PCIe functions that can be discovered, managed, and manipulated like any other PCIe device, and therefore can be used to configure and control PCIe devices (e.g., queues 202 to 208). Each physical function 212 to 218 may have some number of virtual functions associated with it. It should be noted that the controller 115 may also manage virtual functions, virtual controllers, or the like to present themselves as physical functions and physical controllers to other devices, such as the host system 120, connected to the PCIe bus 210 by virtue of the physical functions 212 to 218 associated with each queue 202 to 208. A virtual function may be a lightweight PCIe function that shares one or more resources with a physical function and a virtual function associated with the physical function. For example, after SR-IOV is enabled in a physical function, the PCI configuration space of each virtual function may be accessed by the bus, device, and function (BDF) number of the physical function. Each virtual function may have a PCI memory space for mapping its register set. The virtual function device driver operates on the register set to enable its functionality, and the virtual function appears as an actual PCIe device that can be accessed by the host system 120 through the PCIe bus 210.

[0028] It should be noted that although Figure 2Four queues 202-208 and four corresponding physical functions 212-218 are illustrated, but in other embodiments, there may be any other number of queues 202-208, each with a corresponding physical function. Each of the queues 202-208 stores commands for a corresponding application on the host system 120, the commands being related to storage access operations of corresponding portions of the underlying memory components 112A-112N associated with the each of the queues 202-208. For example, the queue 208 may receive commands (e.g., data access requests) from the host system 120 via the PF 218 over the PCIe bus 210, including a request to read, write, or erase data in a first portion of the memory component 112A. The arbiter component 113 may arbitrate between other commands stored in the other queues 202 to 206, and when a command in the queue 208 of the PF 218 is selected, the arbiter component 113 may issue a command to the controller 115 to perform the requested memory access operation on the data stored at the identified address in the first portion, and return the requested data and / or a confirmation or error message to the host system 120 as appropriate. The arbiter component 113 may function in the same or similar manner for data access requests from the other queues 202 to 206. As described above, the arbiter component 113 associates one of the physical functions 212 to 218 with each of the queues 202 to 208 so as to allow the underlying memory components 112A to 112N to appear as separate storage devices on the PCIe bus 210.

[0029] Furthermore, each physical function 212-218 can be implemented in privileged mode or normal mode. When implemented in privileged mode, the physical function has a single management point that can control resource manipulation and storage provisioning for other functions implemented in normal mode. Additionally, the physical function in privileged mode can perform management options including, for example, enabling / disabling multiple physical functions, storage and quality of service (QoS) provisioning, firmware and controller updates, vendor unique statistics and events, diagnostics, secure erase / encryption, etc.

[0030] In some embodiments, the host system 120 may run multiple virtual machines 232, 234, 236 by executing a software layer 224 (commonly referred to as a "hypervisor") on the hardware and below the virtual machines, such as in Figure 2224 is schematically shown in FIG. In one illustrative example, the hypervisor 224 may be a component of a host operating system 222 executed by the host system 120. Alternatively, the hypervisor 224 may be provided by an application running under the host operating system 222, or may run directly on the host system 120 without an operating system thereunder. The hypervisor 224 may abstract the physical layer including processors, memory, and I / O devices, and present this abstraction to the virtual machines 232, 234, 236 as virtual devices including virtual processors, virtual memory, and virtual I / O devices. Each of the virtual machines 232, 234, 236 may execute a guest operating system that may utilize underlying virtual devices that may, for example, be mapped to a portion of the memory components 112A to 112N managed by one of the queues 202 to 208 and the arbiter component 113 in the memory subsystem 110. One or more applications may be running on each virtual machine under the guest operating system.

[0031] Each virtual machine 232, 234, 236 may include one or more virtual processors. Processor virtualization may be implemented by scheduling time slots on one or more physical processors through the hypervisor 224 so that from the perspective of the guest operating system, the time slots are scheduled on virtual processors. Memory virtualization may be implemented through a page table (PT), which is a memory structure that translates virtual memory addresses into physical memory addresses. The hypervisor 224 may run at a higher privilege level than the guest operating system, and the guest operating system may run at a higher privilege level than the guest application.

[0032] In one embodiment, there may be multiple partitions on the host system 120 representing virtual machines 232, 234, 236. The parent partition corresponding to virtual machine 232 is a root partition (i.e., root ring 0), which has the additional privilege of controlling the life cycle of other child partitions (i.e., conventional ring 0) corresponding to virtual machines 234 and 236, for example. Each partition has a corresponding virtual memory, and instead of presenting a virtual device, the child partitions see the physical devices assigned to them. When the host system 120 is initially started, the parent partition can see all physical devices, and via a pass through mechanism (e.g., PCIe pass through or direct device assignment), the parent partition can assign NVME devices to child partitions. Physical functions 212 to 218 and associated queues 202 to 208 can appear as separate physical storage resources for each of the virtual machines 232, 234, 236, where the running guest operating system or guest application can access the physical storage resources. In one embodiment, for example, virtual machine 232 is associated with physical function 212, virtual machine 234 is associated with physical function 214, and virtual machine 236 is associated with physical function 216. In other embodiments, one virtual machine may be associated with two or more physical functions, and / or two or more virtual machines may be associated with one physical function. Virtual machines 232, 234, 236 may use corresponding bus, device, and function (BDF) numbers to identify the associated physical functions.

[0033] Figure 3 1 illustrates multiple queues 202-208, an arbiter component 113, and a single media controller 313 in a memory subsystem according to some embodiments of the present disclosure. As described above, the media controller 313 may map each PF 212-218 in the memory subsystem 110 to a corresponding portion of one of the memory components 112A-112N. Figure 31, physical function 212 is mapped to portion 302 of memory component 112A, physical function 214 is mapped to portion 304 of memory component 112A, physical function 216 is mapped to portion 306 of memory component 112N, and physical function 218 is mapped to portion 308 of memory component 112N. In some embodiments, one physical function may be mapped to multiple memory components, and partitioning may be employed in each memory component. For example, physical function 212 (PF0) may be mapped to portion 302 (portion 0) of memory component 112A, portion 306 of memory component 112N, and the like. It should be noted that the mapping between physical functions 212 to 218 may change over time. In other embodiments, the respective portions may span two or more of memory components 112A to 112N. Each portion 302 to 308 may have a fixed size or may have different sizes. For example, portion 302 may be larger than portion 304, portion 304 may be larger than portion 306, and portion 306 may be the same size as portion 308. In one embodiment, each portion 302 to 308 is represented by a unique namespace. A namespace is the number of memory components that can be formatted as logical blocks when one or more memory components 112A to 112N are configured with the NVMe protocol. The NVMe protocol provides access to a namespace that appears as a standard block device on which file systems and applications can be deployed without any modification. Each of the physical functions 212 to 218 may have one or more separate namespaces, each of which is identified by a unique namespace ID (NSID). In addition, there may be one or more shared namespaces that include multiple portions 302 to 308 that can be accessed by two or more of the physical functions 212 to 218.

[0034] In addition, each physical function 212 to 218 may also have an associated number of queue pairs for assigning portions 302 to 308 of memory. The memory subsystem 110 may have a fixed number of input / output (I / O) queue pairs and management queue pairs, which may be distributed among the queues 202 to 208. The I / O queue pairs are used to receive memory access requests (or data commands) from the host system 120, and the management queue pairs are used to receive management commands. The number of I / O queue pairs assigned to each physical function 212 to 218 controls how many concurrent memory access requests can be received at the corresponding memory subsystem 110. For example, if the media controller 312 assigns ten I / O queue pairs, the memory subsystem 110 may receive up to ten memory access requests from the host system 120 before rejecting subsequent requests. While processing the current request, the memory access request remains in the associated I / O queue pair. Once the media controller 313 has completed processing the current request, the next request may be retrieved from the associated I / O queue pair for processing, and another memory access request may be added to the queue.

[0035] Despite Figure 3 1. Although not illustrated, the memory subsystem 110 may include local memory for implementing the queues 202-208, as well as other memory for operations performed by the media controller 313 or other components of the memory subsystem 110. This local memory is separate from the memory components 112A-112N. The local memory may be DRAM, SRAM, or other types of technology, while the memory components 112A-112N may be flash memory or other types of non-volatile memory. The outstanding command queues 202-208 may be first-in, first-out (FIFO) queues.

[0036] The arbiter component 113 may include arbitration logic, a hardware state machine, or the like that determines the order in which requests and / or fill operations are to be processed by the media controller 312. The arbitration logic may specify that requests and / or fill operations are scheduled in the order in which the operations are received. One purpose of the arbitration logic may be to allow the host system 120 to define appropriate shares to govern how commands in the queue should be arbitrated in a dynamic manner. Additional functionality of the arbiter component 113 is discussed below.

[0037] Figure 4A Illustrate multiple queues, arbiter components, and a single media controller in a memory subsystem according to some embodiments of the present disclosure. Memory system 400 includes one or more memory devices 412, such as illustrated by two memory devices 412A and 412B. Memory system 400 also includes a system controller 401 (SSD controller) operatively coupled to memory devices 412A and 412B. System controller 401 includes a media controller 413, an arbiter component 113 (e.g., an arbiter circuit), and multiple queues 402 to 408. Each of queues 402 to 408 is associated with a physical function presented to a host computing system, host system 120, through an interconnect (e.g., a PCIe interface). System controller 401 can present two or more physical functions (PFs) to host system 120 through a PCIe interface. Host system 120 can assign each of the PFs to individual applications, virtual machines, or the like. The host system 120 may request the number of PFs required, and the system controller 401 may assign or otherwise associate one of the queues 402-408 to each of the various PFs. In some embodiments, each of the PFs corresponds to a virtual memory controller associated with a different portion of the memory devices 412A and 412B.

[0038] During operation, the arbiter component 113 (arbiter circuit) may determine the difference between the quota of commands and the count of commands issued from the corresponding queues (402 to 408) to the media controller 413. For example, the arbiter component 113 may generate a vector, list, or other data structure with different values ​​for each of the PFs. For example, the data structure may include a first element in the list corresponding to the first PF (PF0), a second element in the list corresponding to the second PF (PF1), and so on until the nth PF (PFn-1). As described herein, the arbiter component 113 may determine the quota of commands for each of the PFs based on the shares (e.g., allocations or percentages of commands) for the corresponding PFs, the shares being specified by the host system 120. For example, if there are four PFs, the host system 120 may assign a 10% share to the first PF, a 20% share to the second PF, a 30% share to the third PF, and a 40% share to the fourth PF. Alternatively, other shares may be specified for two or more PFs. Assuming PF 0 to PF n-1, the following equation (1) shows how the host system 120 can specify shares.

[0039] {p0,p1,…,pn-1}={10%, 20%, 30%, 40%}

[0040] The arbiter component 113 may determine for each of the PFs whether the corresponding queue has no commands. For example, the queue may include a bit to indicate that it is empty. Alternatively, the arbiter component 113 may use other mechanisms or techniques to determine the empty state. The arbiter component 113 may determine a subset of queues from queues 402 to 408 for arbitration by excluding queues that have no commands and queues that have negative numbers for corresponding difference values. That is, when a PF is empty, the arbiter component 113 may remove the corresponding PF from the data structure. Similarly, the arbiter 113 may remove the corresponding PF with a negative difference value.

[0041] Once the arbitrator component 113 has determined the subset of queues, the arbitrator component 113 may randomly select a selected queue from the subset and issue a command from the selected queue to the media controller 413. Once a queue is selected, the arbitrator component 113 may update the count of the corresponding queue. For example, the arbitrator component 113 may update a counter associated with the selected queue.

[0042] In further embodiments, the arbiter component 113 may store a number(s) of available command slots in the timing window. For example, the number of available command slots may be 100. For each of the PFs, the arbiter component 113 may determine a respective quota of commands based on the respective shares specified by the host system 120 and the number(s) of available command slots. Continuing with the above example with four PFs, the quota for the first PF is 10, the quota for the second PF is 20, the quota for the third PF is 30, and the quota for the fourth PF is 40. The quotas may be expressed in the following equation (2).

[0043] q=s*p

[0044] Where p is the corresponding share. The quota for the list of PFs can be expressed in equation (3).

[0045] {q0, q1,..., qn-1}=s*{p0, p1,..., pn-1}={10, 20, 30, 40}

[0046] Where q is the corresponding quota for PF.

[0047] In some embodiments, the arbiter component 113 includes multiple counters, such as one counter per PF. Alternatively, the arbiter component 113 includes multiple counters per PF with multiple sub-counters to create a dynamic counter, as described below with respect to Figure 6 Description. In one embodiment, each counter is associated with one of the queues 202 to 208. When a command is issued from the corresponding queue to the media controller 413, the arbiter component 113 tracks the count of commands issued from the corresponding queue by incrementing the corresponding counter. In another embodiment, each of the queues 202 to 208 is associated with a plurality of sub-counters, wherein the number of sub-counters corresponds to the number of sub-windows of the timing window. For example, there may be four sub-windows in a timing window having a number of available command slots. In the case where the timing window has 100 available command slots (e.g., s=100), each sub-window has 25 available command slots. The count (or counter) for the list of PFs may be expressed in equation (4).

[0048] {c0,c1,...,cn-1}={15, 18, 25, 30}

[0049] Where c is a counter for each PF port.

[0050] In one embodiment, the arbiter component 113 may randomly select the selected queue from the subset by: calculating the sum of the difference values ​​for each of the queues in the subset; assigning a proportional number of index values ​​to each of the queues in the subset based on a ratio of the respective difference values ​​and the sum; and randomly selecting an index value between one and the value of the sum that specifies the selection of the selected queue from the subset. For example, the arbiter component 113 may implement an arbitration algorithm to always select a PF for each available command slot in the timing window as follows:

[0051] Calculate the difference vector: {q 0 -c 0 ,…q n-1 -c n-1}={-5,+2,+5,+10}

[0052] The subset is preprocessed by:

[0053] Remove PFs with empty cmds in queues from the subset of candidate PFs (e.g., PF2 may not have any commands in its corresponding queue);

[0054] If there is at least one positive PF, then the PF with a negative difference value (qc) is removed (eg, PF0 since it is -5).

[0055] Arbitration is performed only among a subset (e.g., {PF1, PF3}) via the weighted round-robin selection process described below;

[0056] Otherwise, if there is no positive PF, arbitration is performed among the subset containing negative differences.

[0057] Weighted Round Robin Selection Process:

[0058] The selected cohort is randomly selected by:

[0059] Add those differences in the subset (e.g., PF1+PF3=12);

[0060] A number between 1 and the sum is randomly selected (e.g., selected from 1 to 12) to select a PF;

[0061] Updates the counter for this PF.

[0062] In other embodiments, the arbitrator component 113 may randomly select the selected queue from the subset by using a weighted round-robin selection scheme or process. For example, the arbitrator component 113 may randomly select the selected PF from the subset of candidate PFs using a weighted round-robin process, where a number between one and the sum of the difference values ​​is selected to select the selected PF from the subset of candidate PFs. Alternatively, other selection schemes may be implemented to arbitrate from a subset of queues in a random or pseudo-random manner. In one embodiment, the arbitrator component 113 may determine that there are no remaining queues in the subset of queues after excluding queues with no commands and queues with negative numbers for corresponding difference values. In this case, the arbitrator component 113 may determine a second subset of queues by excluding only queues with no commands. In essence, the second subset of queues will include queues with negative difference values.

[0063] In the following example, there are four PFs with the following shares and corresponding quotas, assuming the time window (s) is 100:

[0064] {p0, p1, ..., pn-1} = {10%, 20%, 30%, 40%}

[0065] {q0, q1,..., qn-1}=s*{p0, p1,..., pn-1}={10, 20, 30, 40}

[0066] Assuming all four PFs have the command, the arbitration result may be as follows:

[0067] PFs:{99,200,320,381}

[0068] The selected PF instances for each loop are shown below:

[0069] [2,3,3,1,3,0,2,3,3,3,3,3,1,2,1,2,0,3,3,1,3,2,1,0,1,2,1,1,3,3,0,2,0,2,2,0,3,3,1,2,3,3,1,2,1,2,2,3,3,3,2,3,0,3,3,2,2,3,3,2,3,1,0,1,0,3,3,1,3,2,1,2, 2,2,3,3,3,1,2,3,2,2,1,2,2,3,3,3,3,3,3,3,2,2,1,3,3,3,2,2,0,1,3,1,0,1,2,3,2,3,3,3,2,2,0,2,1,2,3,2,1,2,2,2,3,1,3,3,3,3,2,2,3,1,3,2,3,2,0,3,1,0,2,3,…]

[0070] In another embodiment, a memory system including one or more memory devices and a system controller is operatively coupled with the one or more memory devices to present a plurality of PFs to a host computing system via an interconnect. Each of the PFs has a queue to store commands from the host computing system. For each of the PFs, the system controller determines a quota of commands based on a corresponding share percentage for the corresponding PF, the share being specified by the host computing system. The system controller tracks a current count of commands issued from the corresponding queue and determines a difference between the quota and the current count. The system controller generates a subset of candidate PFs for selecting commands from the corresponding queue by removing any of the PFs whose corresponding queues do not store commands from the host computing system and by removing any of the PFs whose difference is less than zero. The system controller randomly selects a selected PF from the subset of candidate PFs using a weighted round-robin selection process, wherein a number between one and the sum of the differences in the subset is randomly selected to select the selected PF from the subset of candidate PFs. The system controller processes the command from the queue corresponding to the selected PF, and the system controller updates the count of the counter of the selected PF.

[0071] In some embodiments, the system controller receives the share percentage for each of the PFs from the host computing system.Alternatively, the system controller receives control information from the host computing system using one of the PFs or using a sideband communication channel.

[0072] In one embodiment, a system controller includes a single media controller, a plurality of queues, wherein each queue is associated with one of the f PFs, and an arbiter circuit coupled to the queues and the single media controller. For each of the PFs, the arbiter circuit determines a quota of commands based on a respective share percentage for the respective PF, the share percentage being specified by the host computing system; tracks a current count of commands issued from the respective queue, and determines a difference between the quota and the current count. In a further embodiment, the arbiter circuit generates a subset of candidate PFs and randomly selects a selected PF from the subset of candidate PFs using a weighted round-robin selection process. Once selected, the arbiter circuit issues, schedules, or otherwise causes commands from the corresponding queue to be processed. For example, the arbiter circuit issues commands from the queue corresponding to the selected PF to the media controller.

[0073] In a further embodiment, the arbiter circuit stores a number of available command time slots in the timing window. For each of the PFs, the arbiter circuit determines a corresponding quota of commands based on a corresponding share specified by the host computing system and the number of available command time slots. In another embodiment, the arbiter circuit includes a plurality of counters, such as one counter per queue (or per PF). When commands are issued from a corresponding queue to the media controller, the arbiter circuit tracks the count of commands issued from the corresponding queue by incrementing the corresponding counter. Alternatively, the arbiter circuit includes a plurality of sub-counters for each counter per queue. In one embodiment, the number of sub-counters corresponds to the number of sub-windows in the timing window having a number of available command time slots.

[0074] In another embodiment, the arbiter circuit may randomly select a selected queue from a subset by: calculating a sum of the difference values ​​for each of the queues in the subset; assigning a proportional number of index values ​​to each of the queues in the subset based on a ratio of the corresponding difference values ​​and the sum; and randomly selecting an index value between one and the sum to select the selected queue from the subset.

[0075] In one embodiment, the timing window may be used by the arbiter circuit to define a static data structure corresponding to the PF quota and dynamic counter as described in the following data structure. Using the static data structure and the dynamic data structure, the arbiter circuit may implement an algorithm as described below:

[0076] Data Structure

[0077] Static Data Structures : Timing Window => PF Quota:

[0078] Total available command time slots: s = 100

[0079] Assume PF 0~PF n-1,

[0080] {p 0 ,p 1 ,…,p n-1 ={10%,20%,30%,40%}

[0081] quota:

[0082] {q 0 ,q 1 ,…,q n-1}=s*{p 0 ,p 1 ,…,p n-1}

[0083] ={10,20,30,40}

[0084] Dynamic Data Structures :

[0085] Counters for each PF port:

[0086] {c 0 ,c 1 ,…,c n-1}={15,18,25,30}

[0087] Algorithm - Arbitration (PF is always selected):

[0088] Difference:

[0089] {p 0 -c 0 ,…p n-1 -c n-1}={-5,+2,+5,+10}

[0090] Preprocessing:

[0091] Remove PFs with empty cmds in the queue, such as PF2

[0092] If there is a positive PF,

[0093] Then remove the PF with negative qc, PF0

[0094] Arbitration only among {PF1,PF3}

[0095] Otherwise the service has a negative PF with unfinished cmds, PF0

[0096] Random Selection:

[0097] PF1+PF3=12

[0098] Flip a coin to choose from 1 to 12 => then choose PF

[0099] Update the counter of this PF

[0100] As described herein, the arbitrator component 113 may be implemented in hardware as described below with respect to Figure 4B The arbiter circuit implementation described.

[0101] Figure 4B A block diagram illustrating an arbiter circuit 450 according to some embodiments of the present disclosure. The arbiter circuit 450 includes a difference calculation block 452 for each of n ports, each of which corresponds to one of the n PFs. Each difference calculation block 452 calculates the difference (q 0 -c 0). The output of each difference calculation block 452 is input into the candidate port selection block 454 for weight calculation. In addition to the difference value input, the candidate port selection 454 receives an indication of whether the corresponding queue has no commands. For example, the candidate port selection block 454 may receive a signal indicating that a particular port has a job (PF_0_has_job?). As described above, if a particular queue has no commands, then the queue is not considered for arbitration (i.e., not part of the subset). For queues determined to be part of the subset, the candidate port selection 454 outputs the corresponding weight values ​​(PF_0_weight, PF_1_weight, etc.) assigned to the specific PF in the subset. The weighted polling block 456 receives weights from specific PFs in the subset (having commands), selects one of the PFs, and outputs a PF number 458 (e.g., PF# or pf_i) corresponding to the selected PF. The jobs (e.g., commands) stored in the corresponding queues of the selected PFs as indicated by the PF number 458 are submitted to the media controller for processing. Weighted round robin block 456 also updates the count (c) of the corresponding difference calculation block 452 corresponding to the output PF number 458 to adjust further arbitration of PFs. Arbiter circuit 450 may be one implementation of arbiter component 113. Alternatively, other circuits may be used to arbitrate commands from multiple queues.

[0102] In some embodiments, the arbiter circuit may operate according to the following three objectives: 1) Fairness: when all PFs have commands, serve them according to their share; 2) Opportunity: make full use of bandwidth when some PFs are idle; 3) Dispersion: distribute the commands of the PFs into a timing window, trying to prevent command clustering of a single PF. The commands may also be arbitrated using a dynamic counter with a sliding window, as described below with respect to Figure 5 describe.

[0103] Figure 5 A sliding window 500 of multiple sub-counters of a dynamic counter of an arbiter component in a memory subsystem is illustrated according to some embodiments of the present disclosure. As described herein, the arbiter component 113 may include a counter that tracks the current count of commands issued from a particular queue. As shown in Figure 6, the counter may be a dynamic counter with multiple sub-counters for one queue. The number of sub-counters may correspond to the number of sub-windows in a timing window with a number of available command time slots. As illustrated, the first timing window 510 includes four sub-windows 502 to 508. Four sub-counters may be used for the four sub-windows 502 to 508. The first sub-counter counts the number of commands issued from the corresponding queue during the first sub-window 502 (sub-window 0), the second sub-counter counts the number of commands issued from the corresponding queue during the second sub-window 504 (sub-window 1), the third sub-counter counts the number of commands issued from the corresponding queue during the third sub-window 506 (sub-window 2), and the fourth sub-counter counts the number of commands issued from the corresponding queue during the fourth sub-window 508 (sub-window 3). After the first timing window 510, the first sub-counter corresponding to the first sub-window 502 may be reset, and the number of commands issued from the corresponding queue during the fifth sub-window 512 (sub-window 4) may be counted. Therefore, the dynamic counter only resets one of the sub-counters after each sub-window, thereby avoiding restarting the counter from the beginning in each timing window. In other words, the timing window 510 becomes a sliding window. The second timing window 514 can be said to start after the first sub-window 502 (sub-window 0) and pass through the fifth sub-window 512 (sub-window 4). Similarly, the third timing window will start after the second sub-window 504 (sub-window 1).

[0104] It should be noted that, as described above, the arbiter component 113 uses counts within the sliding window when calculating the difference. That is, during the second timing window 514, the arbiter component 113 does not use counts that occurred during the sub-window 502 (sub-window 0) because the corresponding sub-counter is reset and does not track counts during the fifth sub-window 512 (sub-window 4). The sliding window of dynamic counters can produce a smooth output from the arbiter component 113 as an attempt to meet the quota for the corresponding queue within the sliding window.

[0105] Figure 6A physical controller implementing an arbiter component according to some embodiments of the present disclosure is illustrated. As described above, the controller 115 includes an arbiter component 113 and queues 202-208. The queues 202-208 are coupled to a PCIe port 605, which enables communication with the host system 120 across the PCIe bus 210. In one embodiment, the controller 115 further includes a media controller 620. The media controller 620 manages the operation of the storage media in the memory subsystem 110, which includes memory components 112A-112N and optional volatile memory, such as one or more dynamic random access memory (DRAM) devices 610. For example, when the queues 202-208 receive multiple commands to access data in one of the memory components 112A-112N, the arbiter component 113 arbitrates and provides the request to the media controller 620, which retrieves and returns the requested data, writes the requested data, or erases the requested data from the memory components 112A-112N.

[0106] In one embodiment, the controller 115 further includes a quality of service (QoS) module 622 and a sideband management (SM) bus 624. QoS can implement individual quality of service management for each of the queues 202 to 208 of the multiple PFs. When a large storage device, such as one of the memory components 112A to 112N, is divided into smaller partitions, each of which is controlled by a PF, and each of which can be used by different clients (e.g., virtual machines on the host system 120), it can be beneficial to associate QoS characteristics with each individual partition. To meet these requirements, the QoS module 622 attaches QoS controls to each PF. The QoS controls may include, for example, individual storage partition size, bandwidth, or other characteristics. The QoS module 622 can monitor the performance of the queues 202 to 208 over time, and can reconfigure resource assignments as needed to ensure that QoS requirements are met.

[0107] The SM bus 624 communicates with a platform manager (not shown), such as a baseboard management controller (BMC) in the memory subsystem 110, to interface with the platform hardware. The platform manager may interface with different types of sensors built into the memory subsystem 110 to report parameters such as temperature, cooling fan speed, power status, operating system status, etc. The platform manager monitors these sensors and may generate an alarm if any of the parameters are not within certain preset limits, indicating a potential failure. The controller 115 may receive these alarms from the platform manager and / or provide status information to the platform manager via the SM bus 624.

[0108] Figure 7700 is a flow chart of an example method 700 for arbitrating commands from multiple queues associated with multiple physical functions in a memory subsystem according to some embodiments of the present disclosure. The method 700 may be performed by processing logic, which may include hardware (e.g., a processing device, a circuit system, a dedicated logic, a programmable logic, a microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executed on a processing device), or a combination thereof. In some embodiments, the method 700 is performed by Figure 1 The arbitrator component 113 of the embodiment of the present invention is executed. Although shown in a specific sequence or order, unless otherwise specified, the order of the processes may be modified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. In addition, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are possible.

[0109] At operation 710, a processing device determines a quota of commands presented to each PF of a host computing system through an interconnect between the host computing system and a system controller. Each of the plurality of PFs includes a queue to store commands from the host computing system. At operation 720, processing logic determines a difference between the quota of commands and a count of commands previously selected from a corresponding queue for each of the plurality of PFs. At operation 730, processing logic determines whether any of the plurality of queues has no commands. At operation 740, processing logic determines a subset of queues from the plurality of queues by excluding any queues that have no commands and any queues that have a negative number for a corresponding difference (also referred to herein as a negative difference). At operation 750, processing logic randomly selects a selected queue from the subset. At operation 760, processing logic processes commands from the queue corresponding to the selected PF.

[0110] In a further embodiment, the processing logic stores the number of available command time slots in the timing window. The processing logic may receive a designated share for each of the plurality of PFs from the host computing system. The processing logic may determine a quota of commands for each of the plurality of PFs, including determining the quota of commands based on the respective designated share and the number of available command time slots.

[0111] In a further embodiment, the arbiter logic may randomly select the selected queue by: calculating the sum of the difference values ​​for each of the queues in the subset; assigning a proportional number of index values ​​to each of the queues in the subset based on the ratio of the corresponding difference values ​​and the sum; and randomly selecting an index value between one and the sum to select the selected queue from the subset.

[0112] In further embodiments, processing logic may determine that a subset of queues is empty after excluding queues with empty commands and queues with negative numbers for corresponding difference values. Thus, processing logic may determine a subset of queues from the plurality of queues by excluding only queues with no commands. That is, the subset of queues may include PFs with negative difference values.

[0113] Figure 8 An example machine illustrating a computer system 800 within which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed. In some embodiments, the computer system 800 may correspond to a host system (e.g., Figure 1 ) that includes, is coupled to, or utilizes a memory subsystem (e.g., Figure 1 110), or may be used to execute operations of the controller (eg, execute an operating system to execute operations corresponding to Figure 1 In some embodiments, the machine may be connected (e.g., using a network) to other machines in a LAN, an intranet, an extranet, and / or the Internet. The machine may operate in the capacity of a server or a client user machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client user machine in a cloud computing infrastructure or environment.

[0114] The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a network appliance, a server, a network router, a switch or a bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by the machine. Further, while a single machine is described, the term "machine" shall also be taken to include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0115] The example computer system 800 includes a processing device 802, a main memory 804 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM), such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), etc.), a static memory 806 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 818, which communicate with each other via a bus 830.

[0116] The processing device 802 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, the processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets, or a processor that implements a combination of instruction sets. The processing device 802 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The processing device 802 is configured to execute instructions 826 for performing the operations and steps discussed herein. The computer system 800 may further include a network interface device 808 for communicating over a network 820.

[0117] The data storage system 818 may include a machine-readable storage medium 824 (also referred to as a computer-readable medium) on which is stored one or more sets of instructions 826 or software embodying any one or more of the methodologies or functions described herein. During execution of the instructions 826 by the computer system 800, the instructions 826 may also reside, in whole or in part, within the main memory 804 and / or within the processing device 802, the main memory 804 and the processing device 802 also constituting machine-readable storage media. The machine-readable storage medium 824, the data storage system 818, and / or the main memory 804 may correspond to Figure 1 Memory subsystem 110.

[0118] In one embodiment, instructions 826 include instructions to implement a method corresponding to a cache component (e.g., Figure 1 113). Although the machine-readable storage medium 824 is shown as a single medium in the example embodiment, the term "machine-readable storage medium" should be considered to include a single medium or multiple media storing one or more sets of instructions. The term "machine-readable storage medium" should also be considered to include any medium capable of storing or encoding a set of instructions for execution by a machine and causing the machine to perform any one or more of the methods of the present disclosure. Therefore, the term "machine-readable storage medium" should be considered to include (but not limited to) solid-state memory, optical media, and magnetic media.

[0119] Some portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is generally considered here to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulation of physical quantities. Typically, but not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. It has proven convenient at times, primarily for common sense reasons, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0120] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure may refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) numbers within the computer system's registers and memories into physical quantities similarly represented as within the computer system's memories or registers or other such information storage systems.

[0121] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specifically constructed for the intended purpose, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. This computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs and magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions, each of which is coupled to a computer system bus.

[0122] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general purpose systems may be used with programs according to the teachings herein, or it may prove convenient to construct more specialized equipment to perform the methods. The structures of various these systems will appear as set forth in the description below. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated that the teachings of the present disclosure as described herein may be implemented using various programming languages.

[0123] The present disclosure may be provided as a computer program product or software, which may include a machine-readable medium having instructions stored thereon, which instructions may be used to program a computer system (or other electronic device) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). In some embodiments, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) readable storage medium, such as a read-only memory ("ROM"), a random access memory ("RAM"), a magnetic disk storage medium, an optical storage medium, a flash memory component, etc.

[0124] In the foregoing description, embodiments of the present disclosure have been described with reference to specific example embodiments of the present disclosure. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope of the embodiments of the present disclosure as set forth in the appended claims. Therefore, the description and drawings should be regarded in an illustrative rather than a restrictive sense.

Claims

1. A memory system, comprising: one or more memory devices; Peripheral Component Interconnect Express (PCIe) interface; a plurality of queues, each queue of the plurality of queues being associated with a physical function (PF) of a plurality of PFs presented to a host computing system via the PCIe interface; and an arbiter circuit coupled to the plurality of queues, wherein the arbiter circuit is configured to: for each of the plurality of PFs, determining a difference between a quota of commands and a count of commands issued from a corresponding queue, wherein the quota of commands is based on a share for the corresponding PF, the share being specified by the host computing system; For each of the plurality of PFs, determining whether the corresponding queue has no commands; determining a subset of queues from the plurality of queues by excluding one or more queues without commands and one or more queues having negative numbers for corresponding differences; The selected queue is randomly selected from the subset of queues, wherein the arbiter circuit is configured to: calculating a sum of the differences for each of the queues in the subset of queues; assigning a proportional number of index values ​​to each of the queues in the subset of queues based on a ratio of the respective difference value to the sum; and randomly selecting an index value between one and the value of the sum, the index value specifying selection of the selected queue from the subset of queues; and A command is issued from the selected queue.

2. The memory system of claim 1 , wherein the arbiter circuit is further configured to: storing a number of available command time slots in a timing window; and For each of the plurality of PFs, a respective quota of commands is determined based on the respective share specified by the host computing system and the number of available command time slots.

3. The memory system of claim 1 , wherein the arbiter circuit further comprises a plurality of counters, each counter being associated with one of the plurality of queues, wherein the arbiter circuit further tracks the count of commands issued from the respective queue by incrementing the respective counter when a command is issued from the respective queue.

4. The memory system of claim 1 , wherein the arbiter circuit comprises a plurality of sub-counters for each of the plurality of queues, wherein a number of the plurality of sub-counters corresponds to a number of sub-windows in a timing window having a number of available command slots.

5. The memory system of claim 1, wherein the arbiter circuit uses a weighted round-robin selection process to randomly select the selected queue from the subset of queues.

6. The memory system of claim 1 , wherein the arbiter circuit is further configured to: determining that no queues remain in the subset of queues after excluding the one or more queues without commands and the one or more queues having the negative numbers for the corresponding differences; and A second subset of queues is determined from the plurality of queues by excluding the one or more queues without commands and including the one or more queues having the negative numbers for the respective differences.

7. An apparatus for arbitrating commands from a plurality of queues, comprising: interface; and a system controller coupled to the interface, wherein the system controller is configured to: providing a plurality of physical functions (PFs) to a host computing system through the interface, wherein each of the plurality of PFs is associated with a queue of the plurality of queues to store commands from the host computing system; for each of the plurality of PFs, determining a difference between a current count of commands issued from a corresponding queue and a quota of commands specified by the host computing system; generating a subset of candidate PFs to select commands from a corresponding queue by removing any of the PFs where the corresponding queue does not store commands from the host computing system and removing any of the PFs where the difference value is less than zero; selecting a selected PF from the subset of candidate PFs by calculating a sum of the difference values ​​for each of the subset of candidate PFs, assigning a proportional number of index values ​​to each of the subset of candidate PFs based on a ratio of the respective difference values ​​to the sum, and randomly selecting an index value between one and the value of the sum, the index value specifying the selection of the selected PF; and Commands from the queue corresponding to the selected PF are processed.

8. The apparatus of claim 7, wherein the system controller is configured to determine the quota of commands using respective share percentages for respective PFs, the respective shares being specified by the host computing system, wherein the system controller is further configured to receive the respective share percentages for each of the plurality of PFs from the host computing system.

9. The apparatus of claim 7, wherein the system controller is further for receiving control information from the host computing system using one of the plurality of PFs.

10. The apparatus of claim 7, wherein the system controller comprises: a plurality of queues, each queue of the plurality of queues being associated with one of the plurality of PFs; and an arbiter circuit coupled to the plurality of queues, wherein for each of the plurality of PFs, the arbiter circuit is configured to: determining the quota for the command based on respective share percentages for respective PFs, the respective share percentages being specified by the host computing system, tracking the current count of commands issued from the corresponding queue, and The difference between the quota and the current count is determined.

11. The apparatus of claim 10, wherein the arbiter circuit is further configured to: generating the subset of candidate PFs; randomly selecting the selected PF from the subset of candidate PFs; and The command is issued from the queue corresponding to the selected PF to a media controller.

12. The apparatus of claim 10, wherein the arbiter circuit is further configured to: storing a number of available command time slots in a timing window; and For each of the plurality of PFs, a respective quota percentage for commands is determined based on the respective share specified by the host computing system and the number of available command time slots.

13. The apparatus of claim 10, wherein the arbiter circuit further comprises a plurality of counters, each counter being associated with one of the plurality of queues, wherein the arbiter circuit is further configured to track the count of commands issued from the respective queue by incrementing the respective counter when a command is issued from the respective queue.

14. The apparatus of claim 10, wherein the arbiter circuit comprises a plurality of sub-counters for each of the plurality of queues, wherein a number of the plurality of sub-counters corresponds to a number of sub-windows in a timing window having a number of available command slots.

15. A method for arbitrating commands from a plurality of queues, comprising: determining, by a system controller of a memory system, a quota of commands for each of a plurality of physical functions (PFs) presented to a host computing system via an interconnect between the host computing system and the system controller, wherein each of the plurality of PFs includes a queue of the plurality of queues to store commands from the host computing system; determining, by the system controller, a difference between the quota of commands and a count of commands previously selected from a corresponding queue for each of the plurality of PFs; determining, by the system controller, whether one or more queues are free of commands; determining, by the system controller, a subset of queues by excluding the one or more queues having no commands and one or more queues having negative numbers for corresponding differences; The selected queue is randomly selected by the system controller from the subset of queues: calculating a sum of the differences for each of the queues in the subset of queues; assigning a proportional number of index values ​​to each of the queues in the subset of queues based on a ratio of a respective difference value to the sum; and randomly selecting an index value between one and the sum, the index value specifying selection of the selected queue from the subset of queues; and Commands from the queue corresponding to the selected PF are processed by the system controller.

16. The method according to claim 15, further comprising: storing a number of available command time slots in a timing window; and A designated share is received from the host computing system for each of the plurality of PFs, wherein the determining the quota of commands for each of the plurality of PFs comprises determining the share of commands based on a respective designated share and the number of available command time slots.

17. The method according to claim 15, further comprising: determining that no queues remain in the subset of queues after excluding the one or more queues without commands and the one or more queues having the negative numbers for the corresponding differences; and A second subset of queues is determined by excluding the one or more queues without commands and including the one or more queues having the negative numbers for the corresponding differences.

Images