System and method for resource-based scheduling commands

By synchronizing hardware and software resources in the storage device, ensuring that the commands are processed as expected, the problem of QoS inability to meet due to insufficient hardware resources is solved, and a more stable storage device command processing is achieved.

CN114003369BActive Publication Date: 2024-11-26SAMSUNG ELECTRONICS CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202110849020.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-10-28
Filing Date
2021-07-27
Publication Date
2024-11-26
Estimated Expiration
2041-07-27

AI Technical Summary

Technical Problem

In the prior art, when processing storage device commands, the quality of service (QoS) may not be met due to insufficient hardware resources, resulting in jitter and processing delays.

Method used

By receiving commands and storing them in a queue, obtaining resource information managed by storage devices, synchronizing resources, allocating resources to the pool, determining the allocation policy based on resource availability, and storing the commands in the queue associated with the storage device to ensure the QoS requirements satisfy.

Benefits of technology

Effectively match hardware resources and software resources, ensure that commands are processed within the expected time window, reduce jitter and delay, and meet service quality requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114003369B_ABST
    Figure CN114003369B_ABST
Patent Text Reader

Abstract

A system and method for scheduling commands for processing by a storage device is disclosed. A command is received from an application and stored in a first queue. Information about a first set of resources managed by the storage device is obtained. A second set of resources is synchronized based on the information about the first set of resources. The second set of resources is allocated to a first pool and a second pool. A condition for the second set of resources in the first pool is determined. Based on a first determination of the condition, one of the second set of resources in the first pool is allocated to the command, and based on a second determination of the condition, one of the second set of resources in the second pool is allocated to the command.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 057,771, filed on July 28, 2020, entitled “RESOURCE-BASED SCHEDULI NG FORQOS IN SSDS,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] One or more aspects according to embodiments of the present disclosure relate to a storage device, and more particularly, to a system and method for scheduling commands based on hardware resources of the storage device. Background Art

[0004] The host computing device may provide commands (e.g., read and write commands) to be processed by the storage device. The host computing device may have certain quality of service (QoS) expectations for the processing of commands. However, in some cases, there may not be enough hardware resources to process commands that meet the QoS requirements, resulting in jitter.

[0005] Therefore, there is a need for a system and method for scheduling commands for processing by a storage device that takes into account the availability of hardware resources of the storage device to address QoS requirements. Summary of the invention

[0006] Embodiments of the present disclosure are directed to a method for scheduling commands for processing by a storage device. The method includes: receiving a command from an application and storing in a first queue; obtaining information about a first set of resources managed by the storage device; synchronizing a second set of resources based on the information about the first set of resources; allocating the second set of resources to a first pool and a second pool; determining a condition of the second set of resources in the first pool; based on a first determination of the condition, allocating one of the second set of resources in the first pool to the command; and based on a second determination of the condition, allocating one of the second set of resources in the second pool to the command. Based on allocating one of the second set of resources, the command is stored in a second queue associated with the storage device for processing by the storage device.

[0007] According to one embodiment, the command includes a data operation from an application program running on the host, and processing the command includes performing the data operation with respect to the non-volatile storage medium.

[0008] According to one embodiment, the information about the first set of resources includes availability of the first set of resources for processing the command.

[0009] According to one embodiment, synchronizing the second set of resources includes: determining satisfaction of a criterion; determining a quantity of the available first set of resources based on the satisfaction of the criterion; and setting the quantity of the available second set of resources equal to the quantity of the available first set of resources.

[0010] According to one embodiment, the criteria include expiration of a time period.

[0011] According to one embodiment, the first pool is shared by the first queue and the second queue, and the second pool is reserved for the first queue.

[0012] According to one embodiment, the second pool is dynamically determined based on detected criteria. The detected criteria may include an identification of a quality of service requirement of the first queue.

[0013] According to one embodiment, a method for scheduling commands for processing by a storage device also includes: allocating, by the storage device, one of a first set of resources based on storing the command in a second queue; processing the command by the storage device; and deallocating, based on completion of processing the command, one of the first set of resources.

[0014] According to one embodiment, the condition is the availability of the second set of resources in the first pool.

[0015] Embodiments of the present disclosure also point to a system for scheduling commands for processing by a storage device. The system includes a processor and a memory. The memory stores instructions that, when executed, cause the processor to perform the following operations: receive a command from an application and store it in a first queue; obtain information about a first set of resources managed by a storage device; synchronize a second set of resources based on the information about the first set of resources; allocate the second set of resources to a first pool and a second pool; determine a condition of the second set of resources in the first pool; based on a first determination of the condition, allocate one of the second set of resources in the first pool to the command; and based on a second determination of the condition, allocate one of the second set of resources in the second pool to the command, wherein, based on allocating one of the second set of resources, the command is stored in a second queue associated with the storage device for processing by the storage device.

[0016] These and other features, aspects and advantages of the disclosed embodiments will become better understood with reference to the following detailed description, appended claims and accompanying drawings. Of course, the actual scope of the invention is defined by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0018] Figure 1is a conceptual block diagram of a system for dispatching commands according to current technical mechanisms;

[0019] Figure 2 is a block diagram of a system for resource-based scheduling commands according to one embodiment;

[0020] Figure 3 According to an embodiment Figure 2 Block diagrams of the various abstraction layers of the system;

[0021] Figure 4 is a method for scheduling commands in a first queue and a second queue according to one embodiment Figure 2 A conceptual diagram of

[0022] Figure 5 is a flow chart of a process for resource-based scheduling of commands stored in a software queue, according to one embodiment. DETAILED DESCRIPTION

[0023] Hereinafter, exemplary embodiments will be described in more detail with reference to the accompanying drawings, wherein the same reference numerals represent the same elements throughout. However, the present disclosure may be implemented in various different forms and should not be construed as being limited to the embodiments shown herein. On the contrary, these embodiments are provided as examples so that the present disclosure will be thorough and complete, and the various aspects and features of the present disclosure are fully conveyed to those skilled in the art. Therefore, processes, elements and techniques that are unnecessary for those of ordinary skill in the art to fully understand the various aspects and features of the present disclosure may not be described. Unless otherwise stated, throughout the drawings and written descriptions, the same reference numerals represent the same elements, and therefore their descriptions may not be repeated. In addition, in the drawings, for clarity, the relative sizes of elements, layers and regions may be magnified and / or simplified.

[0024] In general, a storage device may have a limited number of hardware resources for processing commands from a host computing device. The hardware resources may be, for example, hardware queues and associated tokens that may be used to schedule commands for processing. The size of the hardware queues and the associated number of tokens may be implementation specific and may depend on the performance and QoS requirements of the storage device.

[0025] A device driver interacting with a storage device may allocate software resources to commands to be processed. The software resources may be, for example, software queues and associated tokens. The depth of the software queues and associated tokens may be arbitrary and exceed the depth of the hardware queues and their associated tokens. Sometimes, this may result in the device driver scheduling more commands than the hardware resources available in the storage device. In this case, commands scheduled by the device driver but unable to consume hardware resources may experience processing delays, resulting in high jitter.

[0026] Figure 1 1 is a conceptual block diagram of a system for scheduling commands according to current technical mechanisms. The system includes a device driver 100, which is configured to submit commands for processing by a storage device 102. The device driver 100 may include various software queues 104-108 for storing commands submitted by one or more applications of a host processing device. For example, queue 104 (Q1) may be a deep write queue configured to store write commands, while queue 106 (Q2) may be a shallow read queue configured to store read commands. Commands may consume software resources (e.g., software queues 104-108 and tokens 110) to pass commands to the storage device 102 for processing.

[0027] Similar to the device driver 100, the storage device 102 may include one or more hardware queues 112-114 configured to store commands submitted by the device driver 100. For example, the hardware queue 112 may be configured to store commands obtained from Q1 104 and Q2 106. The commands in the hardware queue 112 may consume hardware resources (e.g., hardware tokens) 116. Commands with assigned hardware resources 116 may be scheduled for processing by the storage device.

[0028] Various software queues 104-108 configured in the device driver can be used to address QoS provisions for processing commands. For example, by separating write and read commands into separate queues 104, 106, and by employing a scheduling algorithm such as round robin, read commands in Q2 can generally be serviced in a predictable time window. However, in some cases, although some commands in Q2 are queued and expected to be processed, resource conflicts within the storage device may prevent commands in Q2 from being serviced as expected.

[0029] To illustrate this, consider an example where Q1 104 has 1000 write commands queued and Q2 106 has 10 read commands queued. For the purposes of this example, assume that the 1000 write commands consume 1000 software tokens 110 and can be submitted to the storage device for processing. However, according to this example, the storage device 102 only has 768 hardware tokens 116 to be assigned. Therefore, the storage device 102 obtains up to 768 write commands from Q1 104 for processing, resulting in all 768 hardware tokens 116 being consumed. Assuming that the write commands have consumed all hardware tokens 116, more write commands to which software tokens 110 are assigned are left to be processed, then the read commands in Q2 may not be processed within their expected time window. Therefore, the QoS requirements of Q2 may not be met.

[0030] Embodiments of the present disclosure are directed to a system and method for resource-based scheduling commands, which take into account the availability of hardware resources of a storage device to submit commands for processing. According to one embodiment, the hardware resources of the storage device are monitored to match / synchronize with software resources. Once synchronized, software resources can be assigned to commands as desired. Assigning software resources to commands can be based on, for example, a determined QoS set forth in a service level agreement (SLA). Although QoS is used as an example of a standard that can be used to determine software resource allocation, it will be appreciated by those skilled in the art that other standards may also be considered, such as user preferences, etc.

[0031] According to one embodiment, the device driver includes a software token manager, which is configured to identify hardware resource information and update the software resource information accordingly. In one embodiment, the software token manager updates multiple available software resources to match multiple available hardware resources. A specific portion of the available software resources can be reserved for one or more queues of the device driver based on, for example, QoS expectations. The remaining software resources can be part of a shared pool. In one embodiment, the software token manager assigns software tokens / resources from the shared pool to commands in the software queue to submit the commands to the storage device for processing. According to one embodiment, if there are no software resources in the shared pool to be assigned to the commands in the queue, the software token manager can access the software tokens reserved for the queue and assign one of the reserved tokens to the command. The reserved tokens can allow the command to be processed in a timely manner to meet the QoS requirements.

[0032] Figure 2 is a block diagram of a system for resource-based scheduling commands according to one embodiment. The system may include a host computing device 200 coupled to a storage device 202 via a storage interface bus 204. The storage device 202 may be a non-volatile storage device, such as, for example, a solid-state drive (SSD), an Ethernet SSD (eSSD), a universal serial bus (USB) drive, a secure digital (SD) card, an embedded multimedia controller (eMMC), a universal flash memory (UFS), etc. The storage interface bus 204 may be, for example, a peripheral component interconnect express (PCIe) bus, Ethernet, and CXL (Compute Express Link). In one embodiment, the host computing device 200 transmits data to the storage device 202 and receives data from the storage device via the storage interface bus 204 using a storage interface protocol. The storage interface protocol may be, for example, a non-volatile memory expression (NVMe) protocol or any other similar protocol that uses a queue to store commands to be processed.

[0033] In one embodiment, the host computing device 200 includes one or more applications 206 running in an application layer of the host computing device 200. The one or more applications 206 may be software applications stored in a host memory space for execution by a processor. In one embodiment, the one or more applications 206 may send commands to the storage device 202 for processing. For example, the one or more applications may issue a read command for reading data from the storage device 202, a write command for writing data to the storage device 202, and / or other input / output (I / O) requests.

[0034] In one embodiment, the host computing device 200 includes a device driver 208 configured to interface with the storage device 202. In one embodiment, the device driver 208 is implemented as a software instruction stored in the host memory and executed by the processor. The device driver 208 may include one or more queues 212a, 212b (hereinafter referred to as software queues 212). The software queues 212 may include, for example, one or more submission queues and completion queues. The submission queues may be configured to store commands / requests submitted by various applications 206. The completion queues may be configured to store completion messages for commands / requests processed by the storage device 202.

[0035] One or more software queues 212 (e.g., submission queues) may be dedicated to storing certain types of commands from the host computing device 200. For example, one queue may be dedicated to storing read commands from an application, while another queue may be dedicated to storing write commands. In one embodiment, certain QoS requirements may be imposed on the software queues 212 based on one or more SLAs. For example, a certain number of commands in the software queues 212 may be expected to be processed within a given period of time. Different QoS requirements may be associated with different software queues 212.

[0036] The device driver 208 may also include a token manager 210 configured to manage software resources that may affect the scheduling of commands to be processed. The software resources may be, for example, software tokens and / or queues 212. In one embodiment, the software token manager 210 is configured to determine the availability of software tokens based on information from the storage device 202 and assign available tokens to commands stored in the software queues 212. The software token manager 210 may be configured to maintain availability information on a per-controller and / or token type basis. For commands submitted to a queue, the assigned token may be from the shared pool, or from a pool reserved for the queue, when a token is available in the shared pool.

[0037] In one embodiment, the storage device 202 includes a host interface layer (HIL) 214 for interfacing between the host computing device 200 and the device control subsystem 216. The HIL 214 may include, but is not limited to, one or more controllers 218 and one or more queues 220a-220c (hereinafter referred to as hardware queues 220). The depth of the hardware queue 220 may depend, for example, on the processing capabilities of the storage device 202. In one embodiment, different queues may be maintained for different types of processing of the storage device 202. For example, one hardware queue may store commands for processing by a hardware acceleration engine, and another queue may store commands for processing via firmware.

[0038] The controller 218 may be implemented via one or more processors, such as, for example, a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). One of the one or more controllers 218 may be associated with one or more hardware queues 220. One of the controllers 218 may be configured to manage hardware resources and assign the hardware resources to commands submitted for processing in the software queues 212. The hardware resources may be, for example, hardware tokens and / or queues 220.

[0039] In one embodiment, the one or more controllers 218 are configured to retrieve I / O requests / commands from one or more of the software queues 212 and store the retrieved requests in one or more hardware queues 220 corresponding to the one or more controllers 218. In some embodiments, the request may be retrieved and submitted by the device driver 208.

[0040] A hardware token may be assigned to commands stored in the hardware queues 220 for processing. In one embodiment, different types of hardware tokens may be maintained and assigned depending on the type and number of hardware queues 220. For example, a first type of hardware token may be assigned to a first hardware queue 220a dedicated to processing by a hardware acceleration engine, while a second type of hardware token may be assigned to a second hardware queue 220b dedicated to processing via firmware.

[0041] The number of available hardware tokens to be assigned to commands in different hardware queues 220 may depend on the number of commands already present in the queues. In one embodiment, tokens are assigned by the controller 218 based on the hardware queue type. In one embodiment, hardware tokens are automatically assigned when commands are stored in one or more hardware queues 220. Commands that have been assigned hardware tokens may be scheduled for processing by the device control subsystem 216.

[0042] In one embodiment, the device control subsystem 216 interacts with the controller 218 to execute commands requested by the application 206. The device control subsystem 216 may include, but is not limited to, one or more processors 222 and one or more media interfaces 224. The one or more processors 222 may be configured to execute computer-readable instructions for processing commands to and from the controller 218 and for managing the operation of the storage device 202. The computer-readable instructions executed by the one or more processors 222 may be, for example, firmware code.

[0043] In one example, one or more processors 222 may be configured to interact with controller 218 to receive write or read commands to or from NVM media 226. One or more processors 222 may interact with NVM media 226 over media interface 224 to implement the write or read actions. NVM media 226 may include one or more types of non-volatile memory, such as, for example, flash memory.

[0044] In one embodiment, the storage device 202 also includes an internal memory 228 for short-term storage or temporary storage during operation of the storage device 202. The internal memory 228 may include DRAM (dynamic random access memory), SRAM (static random access memory), and / or DTCM (data tightly coupled memory). The internal memory 228 may be used to store, for example, the hardware queue 220.

[0045] Figure 3 According to an embodiment Figure 2 A block diagram of the various abstraction layers of the system. For ease of explanation, reference will be made to Figure 2 describe Figure 3 The abstraction layer of the host computing device 200 may include an application layer 300 and a device driver layer 302. The application layer 300 may be configured to generate commands based on I / O requests issued by the application 206. The commands generated by the application layer 300 may be acknowledged to a storage interface protocol of the storage interface bus 204 used to communicate with the storage device 202.

[0046] In one embodiment, the device driver layer 302 includes a software queue 212 and a token manager 210. Commands generated by the application layer 300 are stored in the queue 212 in the device driver 208 based on, for example, the command type. The software token manager 210 can determine the availability of software tokens based on information from the storage device 202 and assign available tokens to the commands stored in the software queue 212. In the event that all tokens are assigned, service may be denied. In one embodiment, commands may be allowed to queue in the software queue 212, but if there are no available tokens, the command may not be submitted to the hardware queue 220 for processing.

[0047] In one embodiment, storage device 202 includes storage layer 304 which may include various sub-layers. The various sub-layers may include host interface layer 214 and other layers for interfacing with NVM media 226, such as flash translation layer 306 and flash interface layer 308. NVM media 226 may be included in storage layer 304 as flash media 310.

[0048] Figure 4 is a method for scheduling commands in a first queue and a second queue according to one embodiment Figure 2 The software token manager 210 can manage information about available software tokens 400, including information about a plurality of available software tokens in a shared pool and information about a plurality of available software tokens in a reserved pool. In one embodiment, the software token manager 210 maintains token availability information for queues 112 managed by controllers of respective controllers 218 in the storage device 202. In the case where the storage device 202 includes multiple controllers 218, the software token manager 210 can maintain multiple sets of token availability information for queues managed by the multiple controllers 218.

[0049] The software token manager 210 can also maintain availability information for different types of tokens in a given set of available tokens. For example, different tokens can be assigned based on whether a command is processed by a hardware acceleration engine or via firmware. The software token manager 210 can be configured to monitor the availability of tokens for hardware acceleration processing and tokens for firmware processing, and assign the appropriate type of token to a command based on availability.

[0050] In one embodiment, a certain number of software tokens can be reserved in the reservation pool for each software queue 212. Reservation can also be based on different token types. For different queues and token types, the number of software tokens reserved for the software queues can be the same or different.

[0051] In one embodiment, the number of tokens (which should be understood to also include the type of token) reserved for a queue 212 can be dynamically determined based on one or more criteria. The criteria can be, for example, usage statistics and / or SLA requirements. In one embodiment, the software token manager 210 is configured to identify the SLA requirements of the queue 212 (e.g., required input / output per second, bandwidth, etc.) and dedicate a portion of the available software tokens 400 based on the SLA requirements. For example, if the bandwidth requirement of the queue is BW1, and the total available bandwidth is BW, the number of reserved tokens can be based on the ratio of BW1 to BW (BW1 / BW).

[0052] In some embodiments, the software token manager 210 is configured to collect historical information about the use of the software queues, including time periods of use, amount of use, applications responsible for the use, etc., and dynamically adjust the number of tokens reserved for the queues based on the historical information. In this way, when a certain time period arrives when the usage of one of the queues 212 is predicted to be high, the software token manager 210 can be configured to reserve more tokens for the queue based on the prediction.

[0053] In yet other embodiments, the software token manager 210 receives information from various applications 206 regarding the number and type of commands expected (e.g., in the next millisecond) from the applications 206. The software token manager 210 can dynamically adjust the number of tokens reserved for the affected queues based on the information. For example, if the first queue is expected to be more fully utilized than the second queue, the number of tokens reserved for the first queue in a given time period can be proportionally higher than the number of tokens reserved for the second queue.

[0054] In one example, in a given time window, the total number of available software tokens 400 identified by the software token manager 210 for a particular controller 218 is 768 tokens. The number of available software tokens 400 can match the total number of hardware tokens 402 associated with the particular controller 218. The software token manager 210 can be configured to reserve a certain number of available software tokens 400 in a reserved pool. For example, 10% of the available tokens (77 tokens) can be allocated to the reserved pool, while the remaining 691 tokens can be allocated to the shared pool. In the case where the first 212a (Q1) in the software queue has 691 commands stored in the queue and the second 212b (Q2) in the software queue has two commands stored in the queue, the software token manager 210 can be configured to schedule the commands in Q1 212a for processing based on a scheduling algorithm. The scheduling algorithm is, for example, a scheduling algorithm such as round-robin, weighted round-robin, fixed priority, etc. The commands in Q1 212a may be allocated 691 available software tokens in the shared pool to allow the commands to be submitted to the associated hardware queue 220a for processing. The 691 commands submitted to the hardware queue 220a may consume 691 hardware tokens from the available hardware token pool 402.

[0055] Next, two commands in Q2 212b can be selected for processing according to the scheduling algorithm. Assuming that 691 tokens assigned to the command in Q1 212a have not been released, there will be no more tokens in the shared pool to be assigned to the command in Q2 212b. However, assuming that 77 tokens have been assigned to the reservation pool and divided between each software queue 212, two of the software tokens that have been assigned to Q2 212b in the reservation pool are assigned to the two commands in Q2 212b. In this regard, even if there are no more available software tokens in the shared pool, the two commands in Q2 212b can still be submitted to the corresponding hardware queue 220a to process via the reserved software tokens, allowing the fairness of the maintenance scheduling. The two commands submitted to the hardware queue 220a may consume two tokens from the available hardware tokens, causing the total number of the available hardware tokens 402 to be reduced to 75.

[0056] In one embodiment, information about updated hardware tokens is exposed to the software token manager 210 via commands or via an application programming interface (API) 404. The software token manager 210 can then use the information to synchronize its software tokens 400. Information about updated tokens can be pushed to the software token manager 210 periodically (regularly or irregularly) or retrieved by the software token manager from the storage device 226. For example, based on the processor 222 completing the processing of one or more commands, the controller 218 can submit a completion message to the device driver 208 using an interrupt signal, which in turn can deallocate / vacate one or more corresponding hardware tokens 402. The completion message can be transmitted to the software token manager 210, and / or submitted to a completion queue of one or more software queues 212. When submitted to the completion queue, the software token manager 210 can be configured to periodically obtain the completion message from the completion queue and update the available software tokens 400 accordingly. For example, when the software token manager 210 receives one or more completion messages associated with five processed commands, the software token manager 210 can deallocate the five corresponding software tokens previously assigned to the commands.

[0057] Figure 5 2 is a flowchart of a process for resource-based scheduling of commands stored in software queue 212 according to one embodiment. It should be understood that the order of steps in the process is not fixed, but can be changed to any desired order recognized by those skilled in the art. For ease of explanation, reference will be made to Figure 2 describe Figure 5 .

[0058] The process begins, and at block 500, the software token manager 210 receives a request for a software token for a command in one of the software queues 212. In some embodiments, the placement of a command in a queue may be considered a request for a software token. The request may include an identifier of the queue storing the command and the token type.

[0059] At block 502, the software token manager 210 determines whether information about available software tokens should be updated. The software token may be updated when criteria are met (such as, for example, expiration of a time period) or when it is determined that no software tokens are available in the shared pool and the reserved pool when a request is received.

[0060] If the token availability information is to be updated, the software token manager 210 obtains information about available hardware tokens and synchronizes the available software tokens to match the available hardware tokens. This may include, for example, receiving an interrupt signal from one or more controllers 218 for receiving update information, or querying one or more completion queues for information about commands completed by the storage device 202. In one embodiment, the update information may include token type information, which is used to allow the software token manager 210 to synchronize multiple available software tokens on a per-token type basis.

[0061] At block 506, the software token manager 210 reserves a certain number of available tokens for each queue and each token type. For example, assuming the total number of available tokens is 1000, the software token manager 210 may allocate 10% of the available tokens to the first in the queue and 5% of the available tokens to the second in the queue. The percentages may be static (e.g., manually set), or dynamically determined. Other percentages are also possible. The number of reserved tokens may be based, for example, on the expected usage of the queues and / or the type of queues, QoS expectations, etc.

[0062] At block 508, the remaining available tokens are allocated as shared tokens in a shared pool.

[0063] At block 510, a determination is made as to whether a shared token in the shared pool satisfies a condition. The condition may be, for example, the availability of a shared token to be assigned to the command. If the answer is yes, then at block 512, an available token in the shared pool is assigned to the command. The command may now be submitted / acquired by the controller 218 for processing, resulting in consumption of a hardware token.

[0064] At block 514, the software token manager 210 decreases the number of available software tokens in the shared pool by one.

[0065] Referring again to block 510, if a determination is made that the shared token does not meet the condition (e.g., there is no shared token in the shared pool), then at block 516, the software token manager 210 determines whether there are any tokens reserved for the queue and token type associated with the command. If the answer is yes, then at block 518, an available token in the reserved pool for the queue and token type is allocated to the command.

[0066] At block 520, the software token manager 210 decreases the number of tokens reserved for the queue and token type by one.

[0067] If there is no shared or reserved token to assign to a command, the command may remain in the queue without being serviced until a software token becomes available again.

[0068] In some embodiments, the systems and methods for resource-based scheduling commands discussed above are implemented in one or more processors. The term processor may refer to one or more processors and / or one or more processing cores. One or more processors may be hosted in a single device or distributed across multiple devices (e.g., on a cloud system). The processor may include, for example, an application-specific integrated circuit (ASIC), a general-purpose or dedicated central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), and a programmable logic device such as a field programmable gate array (FPGA). As used herein, in a processor, each function is performed by hardware (i.e., hard-wired) configured to perform the function, or by more general hardware (such as a CPU) configured to execute instructions stored in a non-transient storage medium (e.g., memory). The processor may be manufactured on a single printed circuit board (PCB) or distributed on several interconnected PCBs. The processor may include other processing circuits; for example, the processing circuit may include two processing circuits, an FPGA and a CPU, interconnected on a PCB.

[0069] It should be understood that although the terms "first", "second", "third", etc. may be used to describe various elements, components, regions, layers and / or parts in this article, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or part from another element, component, region, layer or part. Therefore, the first element, component, region, layer or part discussed below can be referred to as a second element, component, region, layer or part without departing from the spirit and scope of the present inventive concept.

[0070] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the inventive concept. As used herein, the terms "substantially," "about," and similar terms are used as approximate terms rather than terms of degree, and are intended to account for the inherent deviations of measured or calculated values ​​that would be recognized by one of ordinary skill in the art.

[0071] As used herein, the singular forms "one" and "an" are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that when used in this specification, the term "comprise / comprising" specifies the presence of the stated features, wholes, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, parts and / or combinations thereof. As used herein, the term "and / or" includes any and all combinations of one or more related listed items. When expressions such as "at least one" are used after a list of elements, the entire list of elements is modified without modifying a single element in the list. In addition, when describing an embodiment of the inventive concept, the use of "may" refers to "one or more embodiments of the present disclosure". In addition, the term "exemplary" is intended to refer to an example or illustration. As used herein, the term "use / using / used" can be considered to be synonymous with the term "utilize / utilizing / utilized", respectively.

[0072] It should be understood that when an element or layer is referred to as being “on,” “connected to,” “coupled to,” or “adjacent to” another element or layer, it may be directly “on,” “connected to,” “coupled to,” or “adjacent to” the other element or layer, or one or more intervening elements or layers may be present. In contrast, when an element or layer is referred to as being “directly on,” “directly connected to,” “directly coupled to,” or “directly adjacent to” another element or layer, there are no intervening elements or layers.

[0073] Any numerical range listed herein is intended to include all subranges of the same numerical precision contained within the listed range. For example, a range of "1.0 to 10.0" is intended to include all subranges between (and including) the listed minimum value of 1.0 and the listed maximum value of 10.0, that is, having a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0, such as 2.4 to 7.6, for example. Any maximum numerical limit listed herein is intended to include all lower numerical limits contained therein, and any minimum numerical limit listed in this specification is intended to include all higher numerical limits contained therein.

[0074] Although exemplary embodiments of systems and methods for resource scheduling commands have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Therefore, it should be understood that, in addition to what is specifically described herein, systems and methods for resource scheduling commands based on principles of the present disclosure may be embodied. The present disclosure is also defined in the appended claims and their equivalents.

Claims

1. A method for scheduling commands for processing by a storage device, the method include: Receives commands from applications and stores them in a software queue; obtaining information about hardware resources managed by the storage device; updating information of software resources based on the information about the hardware resources; Allocating the software resources into a shared pool and a reserved pool, wherein the shared pool is shared by the software queue and the hardware queue, and the reserved pool is reserved for the software queue; determining a condition of the software resource in the shared pool, wherein the condition is an availability of the software resource in the shared pool; allocating one of the software resources in the shared pool to the command based on a first determination that the condition is satisfied; as well as Based on a second determination that the condition is not satisfied, allocating one of the software resources in the reserved pool to the command, Wherein, based on said allocating said one of said software resources, said command is stored in a hardware queue associated with said storage device for processing by said storage device.

2. The method of claim 1, wherein the command comprises a data operation from an application program running on a host, and processing the command comprises performing the data operation with respect to a non-volatile storage medium.

3. The method of claim 1, wherein the information about the hardware resource comprises availability of the hardware resource for processing the command.

4. The method of claim 1, wherein the information of the software resource is updated include: Determine that standards are met; determining an amount of available hardware resources based on satisfaction of the criteria; as well as Set the number of available software resources to be equal to the number of available hardware resources. The method of claim 4 , wherein the criteria includes expiration of a time period. The method of claim 1 , wherein the retention pool is determined dynamically based on detected criteria.

7. The method of claim 6, wherein the criteria for the detection include an identification of a quality of service requirement of the software queue.

8. The method of claim 1, further comprising: include: allocating, by the storage device, one of the hardware resources based on storing the command in the hardware queue; processing the command by the storage device; as well as Based on completion of the processing of the command, the one of the hardware resources is deallocated.

9. A system for scheduling commands for processing by a storage device, the system include: processor; and a memory, wherein the memory stores instructions that, when executed, cause the processor to: Receives commands from applications and stores them in a software queue; obtaining information about hardware resources managed by the storage device; updating information of software resources based on the information about the hardware resources; Allocating the software resources into a shared pool and a reserved pool; determining a condition of the software resource in the shared pool, wherein the condition is an availability of the software resource in the shared pool; allocating one of the software resources in the shared pool to the command based on a first determination that the condition is satisfied; as well as Based on a second determination that the condition is not satisfied, allocating one of the software resources in the reserved pool to the command, Wherein, based on said allocating said one of said software resources, said command is stored in a hardware queue associated with said storage device for processing by said storage device.

10. The system of claim 9, wherein the command comprises a data operation from an application program running on a host, and processing the command comprises performing the data operation with respect to a non-volatile storage medium.

11. The system of claim 9, wherein the information about the hardware resource includes availability of the hardware resource for processing the command.

12. The system of claim 9, wherein the instructions that cause the processor to update information of a software resource include instructions that cause the processor to: Determine that standards are met; determining an amount of available hardware resources based on satisfaction of the criteria; and Set the number of available software resources to be equal to the number of available hardware resources.

13. The system of claim 12, wherein the criteria includes expiration of a time period.

14. The system of claim 9, wherein the retention pool is dynamically determined based on detected criteria.

15. The system of claim 14, wherein the detected criteria include an identification of a quality of service requirement of the software queue.

16. The system of claim 9, wherein the storage device is configured to: allocating one of the hardware resources based on storing the command in the hardware queue; processing the command; and Based on the completion of the processing of the command, a signal is transmitted to deallocate the one of the hardware resources.

Citation Information

Patent Citations

  • Buffer Management in Memory Systems for Read and Write Requests

    US20200050366A1

  • Apparatus and method for bandwidth allocation and quality of service management in a storage device shared by multiple tenants

    US20200089537A1