Submission queue arbitration method and electronic devices for storage devices

Abstract

This application discloses a submission queue arbitration method and electronic device for a storage device, relating to the field of computer storage technology. The method stores instructions sent by the host in multiple submission queues, which are pre-divided into multiple submission queue groups. These groups are further pre-divided into multiple queue sets. In the hierarchical arbitration of queue sets, queue groups, and queues, dynamic arbitration is used instead of a single fixed priority or simple round-robin method. This multi-level queue arbitration method solves the problem in related arbitration mechanisms that cannot finely distinguish the service quality requirements of multi-tenant / multi-service scenarios, lack flexibility, and affect overall service quality. By refining the queue management unit, the scheduling accuracy is improved, enabling more precise fulfillment of service quality requirements for various queues in complex multi-tenant, mixed load scenarios, and avoiding the "starvation" phenomenon of low-priority queues.

Description

Technical Field

[0001] This application relates to the field of computer storage technology, and in particular to a submission queue arbitration method and electronic device for storage devices. Background Technology

[0002] With the widespread adoption of NVMe (Non-volatile Memory Express) storage devices in data centers, scenarios such as multi-tenancy and mixed workloads place higher demands on queue arbitration mechanisms. However, most existing arbitration mechanisms employ a single-priority arbitration method. This method suffers from limitations in resource allocation flexibility and efficiency due to inaccurate priority settings. Furthermore, the single-priority arbitration method cannot precisely differentiate the QoS requirements of multiple tenants / services, lacking flexibility and easily leading to high-priority queue resource monopolies or low-priority queue starvation, thus impacting overall QoS (Quality of Service). Summary of the Invention

[0003] This application provides a submission queue arbitration method and electronic device for storage devices to at least address the problems of inaccurate priority settings, inefficient resource allocation, and insufficient fairness in related technologies.

[0004] This application provides a method for arbitration of commit queues in a storage device, comprising: storing multiple instructions sent by a host into multiple commit queues; the multiple commit queues being pre-divided into multiple commit queue groups; the multiple commit queue groups being pre-divided into multiple queue sets; each queue set in the multiple queue sets corresponding to a first priority; the queue sets in the multiple queue sets being divided according to storage medium type; different storage medium types corresponding to different priorities; the first priority corresponding to each queue set in the multiple queue sets being determined based on the priority corresponding to the storage medium type of each queue set in the multiple queue sets; each commit queue group in the multiple commit queue groups corresponding to a second priority; each commit queue group in the multiple commit queue groups corresponding to multiple namespaces; each namespace corresponding to... Different priorities; the namespaces in the multiple namespaces correspond to different business types and service levels; the priorities of the namespaces in the multiple namespaces are determined according to the business type and service level; the second priority of the submission queue group in the multiple submission queue groups is the highest priority among the multiple namespaces corresponding to the submission queue group in the multiple submission queue groups; the following arbitration process is executed in multiple preset arbitration cycles respectively: according to the first priority of the queue set in the multiple queue sets, the current queue set is selected from the multiple queue sets; according to the second priority of the submission queue group in the multiple submission queue groups, the current submission queue group is selected from the current queue set; and the instructions stored in the current submission queue in the current submission queue group are processed by the processor of the storage device.

[0005] This application also provides a commit queue arbitration device for a storage device, comprising: a queue storage module for storing multiple instructions sent by a host into multiple commit queues; the multiple commit queues are pre-divided into multiple commit queue groups; the multiple commit queue groups are pre-divided into multiple queue sets; the queue sets in the multiple queue sets correspond to a first priority; the queue sets in the multiple queue sets are divided according to storage medium type; different storage medium types correspond to different priorities; the first priority corresponding to the queue sets in the multiple queue sets is determined according to the priority corresponding to the storage medium type of the queue sets in the multiple queue sets; the commit queue groups in the multiple commit queue groups correspond to a second priority; the commit queue groups in the multiple commit queue groups correspond to multiple namespaces; the multiple namespaces correspond to... There are different priorities; the namespaces in the multiple namespaces correspond to different business types and service levels; the priority of the namespaces in the multiple namespaces is determined according to the business type and service level; the second priority of the submission queue group in the multiple submission queue groups is the highest priority among the multiple namespaces corresponding to the submission queue group in the multiple submission queue groups; the queue arbitration module is used to execute the following arbitration process in multiple preset arbitration cycles respectively: select the current queue set from the multiple queue sets according to the first priority of the queue set in the multiple queue sets, select the current submission queue group from the current queue set according to the second priority of the submission queue group in the multiple submission queue groups, and process the instructions stored in the current submission queue in the current submission queue group through the processor of the storage device.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the submission queue arbitration method of any of the above-described storage devices when executing the computer program.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the submission queue arbitration method of any of the above-described storage devices.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the submission queue arbitration method for any of the above-described storage devices.

[0009] This application stores host-sent instructions in multiple commit queues, which are pre-divided into multiple commit queue groups. These commit queue groups are further pre-divided into multiple queue sets, constructing a three-level logical grouping structure: "Queue Set (Set) → Commit Queue Group (Group) → Commit Queue (SQ)". By combining storage media type, service type, and service level, priorities and weights are assigned to different levels, solving the problem of inaccurate priority settings caused by coarse prioritization in related technologies. Through a multi-level nested priority system, precise control over resource allocation at both the basic physical layer (storage media) and the logical layer (service type) is ensured, meeting the stringent QoS requirements of different service and load scenarios. Secondly, the queue sets are pre-divided according to storage media type, with different media type queue sets corresponding to different first priorities, ensuring that critical business and data access operations can first utilize the most suitable storage resources. This approach addresses the problem of inefficient resource allocation. Each submission queue group corresponds to multiple namespaces, each with different business types and service levels. The priority of each namespace is determined based on the business type and service level. Each submission queue group also has a secondary priority, determined by the highest priority among the multiple namespaces associated with that group, ensuring priority processing of critical business processes. In the hierarchical arbitration of queue sets, queue groups, and queues, dynamic arbitration is used instead of a single fixed priority or simple round-robin method. This multi-layered queue arbitration method solves the problems of inaccurate priority settings, inefficient resource allocation, and insufficient fairness in related arbitration mechanisms. By refining the queue management unit, the scheduling precision is improved, enabling more accurate fulfillment of the Quality of Service (QoS) requirements of various queues in complex multi-tenant and mixed load scenarios, avoiding the "starvation" phenomenon of low-priority queues. In summary, the multi-layered arbitration mechanism, through its flexible hierarchical division and independent scheduling strategy, can more precisely meet the QoS requirements of multi-tenant / multi-business environments, avoiding rigid resource allocation and improving overall service quality and user experience. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a hardware structure block diagram of a submission queue arbitration method for a storage device according to an embodiment of this application.

[0012] Figure 2This is a flowchart of a submission queue arbitration method for a storage device according to an embodiment of this application.

[0013] Figure 3 This is a schematic diagram of a layered structure of a storage device provided in an embodiment of this application.

[0014] Figure 4 This is an architecture diagram of a queue grouping architecture according to an embodiment of this application.

[0015] Figure 5 This is a structural diagram of a submission queue arbitration device for a storage device according to an embodiment of this application.

[0016] Figure 6 This is a structural diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0018] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The specific application environment architecture or specific hardware architecture on which the execution of the submission queue arbitration method of the storage device depends is described here.

[0021] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of the submission queue arbitration method for a storage device according to an embodiment of this application. Figure 1 As shown, the server device may include one or more ( Figure 1Only one is shown in the image. A processor 102 (which may include, but is not limited to, a central processing unit (CPU), microprocessor (MCU), or programmable logic device (FPGA), etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0022] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the submission queue arbitration method of the storage device in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0023] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0024] The embodiments of this application provide a submission queue arbitration method for a storage device, and the method is described in detail in conjunction with the execution flow of the submission queue arbitration method for a storage device.

[0025] The following explains the technical terms used in this application:

[0026] SSD: Solid State Disk, also known as a solid-state drive.

[0027] NVMe: Non-volatile Memory Express.

[0028] SLC: Single Level Cell, is a type of flash memory technology where each cell stores 1 bit of data. It features high performance and a long lifespan.

[0029] QLC: Quad Level Cell, is a type of flash memory technology where each cell stores 4 bits of data. It has a large capacity and low cost, but its speed and lifespan are relatively low.

[0030] PCIe: PCI Express, short for Peripheral Component Interconnect High Speed.

[0031] QoS: Quality of Service.

[0032] RR: Round Robin, also known as the round-robin arbitration algorithm.

[0033] RR (Set): Round Robin (Set), also known as the set-based round-robin algorithm.

[0034] WRR: Weighted Round Robin Scheduling, also known as the weighted round-robin scheduling algorithm.

[0035] WRR (SP): Weighted Round Robin (Specific Processing), also known as the specific processing weighted round-robin algorithm.

[0036] This embodiment provides a submission queue arbitration for a storage device. Figure 2 This is a flowchart of the submission queue arbitration of a storage device according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0037] Step S202: Multiple instructions sent by the host are stored in multiple commit queues; multiple commit queues are pre-divided into multiple commit queue groups; multiple commit queue groups are pre-divided into multiple queue sets; each queue set in the multiple queue sets has a first priority; the queue sets in the multiple queue sets are divided according to storage medium type; different storage medium types correspond to different priorities; the first priority of each queue set in the multiple queue sets is determined based on the priority corresponding to the storage medium type of each queue set in the multiple queue sets; each commit queue group in the multiple commit queue groups has a second priority; each commit queue group in the multiple commit queue groups corresponds to multiple namespaces; multiple namespaces correspond to different priorities; each namespace in the multiple namespaces corresponds to different service types and service levels; the priority of each namespace in the multiple namespaces is determined based on the service type and service level; the second priority of each commit queue group in the multiple commit queue groups is the highest priority among the multiple namespaces corresponding to the commit queue group in the multiple commit queue groups.

[0038] Step S204: Execute the following arbitration process within multiple preset arbitration cycles: Select the current queue set from the multiple queue sets according to the first priority corresponding to the queue set in the multiple queue sets; select the current submission queue group from the current queue set according to the second priority corresponding to the submission queue group in the multiple submission queue groups; and process the instructions stored in the current submission queue in the current submission queue group through the processor of the storage device.

[0039] The storage device submission queue arbitration method of this application relates to the field of computer storage technology and is used to optimize queue scheduling efficiency and quality of service (QoS) in high-concurrency scenarios.

[0040] With the widespread adoption of NVMe storage devices in data centers, scenarios such as multi-tenancy and mixed workloads place higher demands on queue arbitration mechanisms. However, most existing arbitration mechanisms employ single-priority or simple round-robin approaches. Single-priority arbitration suffers from inaccurate priority settings, limiting resource allocation flexibility and efficiency. Furthermore, single-priority arbitration can lead to high-priority queues monopolizing resources, starving low-priority queues, and causing serious issues such as host I / O timeouts and application interruptions. Simple round-robin arbitration, on the other hand, can prevent high-priority commands (such as management configuration and fault recovery commands) from being executed first, reducing system response speed and reliability. In summary, existing arbitration mechanisms suffer from inaccurate priority settings, inefficient resource allocation, and insufficient fairness. To address this issue, this application proposes a multi-level arbitration mechanism that uses a hierarchical structure of "Queue Set (Set) → Submission Queue Group (Group) → Submission Queue (SQ)" to finely manage the submission queues (SQ) of NVMe storage devices. In this system, the first priority of the queue set is determined based on the storage medium type to reflect the performance and persistence characteristics of different media. The second priority of the submission queue group is based on the highest priority of the service type and service level of the namespace it manages, ensuring priority processing of critical services. During arbitration, the current queue set is selected from the Set level according to the first priority of the queue set among multiple queue sets. Then, the current submission queue group is selected from the current queue set according to the second priority of the submission queue group among multiple submission queue groups. Finally, the current submission queue is selected from the current submission queue group. Compared with arbitration mechanisms based on single priority or simple round-robin, this embodiment effectively solves the limitations of single priority or simple round-robin and the problem of inaccurate priority settings through a multi-level arbitration mechanism. It achieves precise priority settings and fine-grained differentiation of QoS for multi-tenant and multi-service applications, ensuring rapid response for high-priority tasks while preventing low-priority queues from being "starved".

[0041] In this context, "host" refers to the server or computing device that initiates I / O operations, sending read / write instructions and other management commands to the storage device. Host-initiated instructions are operation requests sent by the host to the storage device, including but not limited to read, write, management configuration, and fault recovery operations, used to instruct the storage device to handle the corresponding operations.

[0042] A commit queue (SQ) is a pre-established queue in a storage device used to receive and store instructions sent by the host. In this embodiment, multiple commit queues are used to receive concurrent I / O instructions of different types from multiple tenants. Different services or tenants typically achieve resource isolation and independent Service Level (QoS) control by creating independent commit queues (SQs). Each SQ can be configured with different priorities and weights to ensure that I / O operations between different services or tenants can be scheduled according to the predetermined service level, avoiding the impact of other low-priority services on critical services.

[0043] A commit queue group is a further division within multiple commit queues. Each commit queue group contains different commit queues to allow for more granular control over resource allocation.

[0044] Multiple queue sets (Set) are the highest level in the arbitration mechanism of this application embodiment. They consist of multiple submission queue groups (Group). Each Set can be configured with priority and weight independently to achieve cross-group resource scheduling.

[0045] In this embodiment, the queue grouping architecture includes a three-level logical grouping of "Set→Group→Group", that is, SQ is divided into three levels of logical groups of "Set→Group→Group". Each level is set with configuration attribute parameters. The Set level sets the Set set parameters required for arbitration submission to the queue. For example, the Set set parameters include configuration priority, preset weight, and skip flag SKIP (used to control whether the current Set participates in queue scheduling). The Group level sets the Group group parameters required for arbitration submission to the queue. For example, the Group group parameters include configuration priority, preset weight, the Set to which the Group belongs (the Set number of the current Group), and skip flag SKIP (used to control whether the current Group participates in queue scheduling). The Queue level sets the Queue queue parameters required for arbitration submission to the queue. For example, the Queue queue parameters include the Enabled configuration status, the Group to which the Queue belongs (the Group to which the current Queue belongs), and the skip flag SKIP that controls whether this queue participates in arbitration.

[0046] Multiple commit queues are pre-divided into multiple commit queue groups, and multiple commit queue groups are pre-divided into multiple queue sets. For example, assuming the storage device has 1025 SQ queues, these 1025 SQ queues are pre-divided into 8 Sets, and each Set is further subdivided into 8 Groups, forming a hierarchical structure of "Set→Group→Group". Specifically, Set1 to Set8 each contain 8 Groups, such as Group1_1 to Group1_8 under Set1. Each Group manages approximately 16 SQ queues, ensuring fine-grained and efficient scheduling control. In this embodiment, the queue sets, commit queue groups, and commit queues constitute a hierarchical structure of a multi-level priority arbitration mechanism, which can effectively manage a large number of concurrent I / O instructions, avoid resource monopolies and "starvation" phenomena, and maintain scheduling fairness while meeting the Quality of Service (QoS) requirements in high-concurrency scenarios.

[0047] In this embodiment, each queue set includes a group of SQs sharing similar storage media types and QoS requirements, used to implement priority scheduling based on storage media type. Here, storage media type refers to different physical media used for data storage in NVMe devices, such as SLC (Single-Level Cell) and QLC (Quad-Level Cell) flash memory. Different media types correspond to different priorities, reflecting their performance characteristics and cost-effectiveness. For example, Figure 3 This is a schematic diagram of a layered structure of a storage device provided in an embodiment of this application, such as... Figure 3 As shown, queue set Set1 uses SLC storage medium, and queue set Set2 uses QLC storage medium. Set1 and Set2 are suitable for storing data with different levels of activity (hot and cold). The queue set using SLC is suitable for storing hot data, offering higher performance and a higher maximum PE count, but with limited capacity. Set2, using QLC, is suitable for storing cold data, offering lower performance and a lower maximum PE count, but with larger capacity. In practical applications, different Set types represent different storage medium type attributes, and different priorities can be pre-assigned according to the media type corresponding to the queue set.

[0048] Each queue set within a set has a corresponding first priority. This first priority corresponds to the priority of the queue set itself, and it is determined based on the type of storage medium the queue set represents. For example, the first priority of a queue set refers to the priority corresponding to the type of storage medium it represents. The set-level priority determines the processing order of which queue sets are processed during the arbitration process. A queue set may contain multiple queue groups, and each queue set has its own priority setting used for priority comparison and scheduling among multiple queue sets.

[0049] Submission queue groups are associated with a specific set of namespaces; that is, a submission queue group is associated with at least one namespace (NS) for more granular management of I / O requests for different service types and service levels. In essence, different queue sets contain different groups, meaning different queue sets contain different host application service types and storage capacity namespaces. A namespace is a logical unit within an NVMe device used to isolate and manage different datasets. Each namespace carries a specific service type and service level, with independent storage space and I / O paths. Different service types provide different QoS service levels. The service type describes the application scenario or functional category served by the data submitted through the namespace, such as real-time transactions, data analysis, and log recording. Different service types have different requirements for I / O performance and response time. The service level reflects the quality standard of the data service provided by the namespace, typically including commitments to transmission rate, latency, and throughput, and is closely related to the QoS requirements of the service type, guiding priority setting.

[0050] Multiple namespaces have different priorities, and the priorities of namespaces within multiple namespaces are predetermined based on business type and service level. For example, weights for business type and service level can be set for each namespace, and the ratio between the weights of the business type and service level can be used to determine the priority of each namespace, or the sum of the weights of the business type and service level can be used to determine the priority of each namespace.

[0051] Each commit queue group has a corresponding secondary priority. This secondary priority is determined based on the highest priority among the multiple namespaces managed within that commit queue group, ensuring that queues for critical business operations receive priority processing.

[0052] Each commit queue group is associated with a set of namespaces, which in turn are associated with the commit queues within that group. In other words, each commit queue in a commit queue group is associated with a namespace within that set of namespaces, meaning there is a one-to-one or many-to-one mapping between each commit queue and a namespace in that set. For example... Figure 3 As shown, in actual applications, different Set types are set with different Group grouping numbers. Each Group can be set with different command space NS number attributes (NS1 / NS2...). During device initialization, it is necessary to establish different SQ submission queues and binding mapping relationships with NS1, NS2... etc., and set different QoS service level types.

[0053] After the first priority determines the set of queues to be processed, the second priority is used to further refine the scheduling, determining which queue groups within the set prioritize scheduling their internal queues. Queues within a queue group may have different characteristics or service levels; by setting the second priority, more granular resource allocation and scheduling strategies can be implemented. Through the above settings of different Set attributes, different Namespace QoS service attribute levels provided by different Group types, and different SQ submission queue priorities, the IO performance isolation and different QoS levels of the entire storage device are achieved.

[0054] Compared to the single priority or simple polling methods in related technologies, in this embodiment, during the arbitration process, the scheduler of the NVMe device first determines which queue set to process based on the first priority, and then selects which queue group to process within that set based on the second priority. This ensures that when processing multiple queue sets and multiple submission queue groups, high-priority queue sets and submission queue groups can be prioritized according to the priority of the service and QoS requirements, avoiding the "starvation" phenomenon and ensuring efficient, fair, and dynamic resource scheduling in complex and ever-changing environments.

[0055] In this embodiment, the preset arbitration period refers to the time window set by the storage device for arbitration queue access. Within each period, the queue arbitration process is executed to determine which submissions to the queue are processed first. The length of the preset arbitration period is crucial for ensuring IOPS performance targets and can be dynamically adjusted according to system requirements. In this embodiment, the length of the preset arbitration period can be dynamically determined based on the overall performance targets and hardware characteristics of the storage system. For example, if the system target is 1M IOPS (input / output operations per second), and the minimum arbitration period allowed by the device hardware is 100ns, then the preset arbitration period can be set to 100ns. The length of the preset arbitration period can also be predictively configured based on business QoS requirements and historical load data. For example, by analyzing the storage device load over the past week, it is found that 9:00 AM to 11:00 AM on weekdays is a high-load period. At this time, the arbitration period can be set shorter (e.g., 100ns) to quickly respond to I / O requests and ensure performance and response speed during peak business periods. At night or during low-load periods, the arbitration period can be appropriately extended (e.g., 200ns) to reduce unnecessary hardware wake-ups and power consumption, while maintaining system stability and efficiency under low load.

[0056] An arbitration process is executed within each preset arbitration cycle. This process includes: at the start of the arbitration cycle, a Set is determined from multiple preset queue sets using a preset filtering mechanism as the current queue set; within the current queue set, a Group is further filtered and determined as the current submission queue group using the preset filtering mechanism; and within the current submission queue group, a specific submission queue is selected as the current submission queue using the preset filtering mechanism. The preset filtering mechanism can be a priority and weight filtering mechanism, i.e., filtering according to the priority and weight of each object to be scheduled; it can also be a load-based dynamic scheduling mechanism, i.e., dynamically selecting objects with relatively low loads based on their load conditions; or it can be a time-slice-based fair scheduling mechanism, i.e., allocating a fixed time slice to each object to ensure that each object has a processing opportunity within a certain time. Other filtering mechanisms are also possible, but will not be elaborated upon here.

[0057] Within each preset arbitration cycle, a single arbitration process or multiple arbitration processes can be executed, depending on performance requirements and the length of the preset arbitration cycle. Executing a single arbitration process within a preset arbitration cycle is suitable for situations where all queues have a chance to be served within a fixed time interval, helping to ensure fairness and avoid "starvation." Executing multiple arbitration processes within a preset arbitration cycle is suitable for scenarios requiring high IOPS (Input / Output Operations Per Second). By increasing the filtering frequency, requests from different queues can be responded to more quickly, improving system throughput and response speed. For example, if the system needs to support extremely high IOPS, it may be necessary to shorten the preset arbitration cycle and increase the number of filtering rounds (i.e., arbitration processes) to shorten the waiting time for each queue to be processed.

[0058] Here, the current queue set refers to the set of queues selected to participate in arbitration decisions within a certain time period. The current commit queue group refers to the group of commit queues selected from the current queue set. The current commit queue refers to the specific queue selected in the current commit queue group, storing host instructions waiting to be processed by a designated core of the processor. A designated core refers to a processor core in a multiprocessor architecture of a storage device that is pre-selected or configured to specifically process instructions from the current commit queues selected from the current queue set and the current commit queue group.

[0059] In some embodiments, if there is a specified submission queue group in the current queue set where none of the submission queues contain instructions, the specified submission queue group is set not to participate in the arbitration process until instructions are stored in the submission queues of the specified submission queue group, thereby controlling the specified submission queue group to participate in the arbitration process.

[0060] The designated commit queue group refers to the commit queue group in the current queue set that does not store any instructions. If a designated commit queue group exists in the current queue set, it will be marked as not participating in the queue scheduling of the current arbitration cycle until at least one SQ begins storing instructions.

[0061] In a set of queues, if there exists a group of commit queues (SQs) where none of them currently store any instructions (i.e., the queues are empty), this group will be set not to participate in the current arbitration process. This is to avoid meaningless comparisons and filtering of empty queues during arbitration, thus saving computational resources and reducing unnecessary arbitration delays. Once any commit queue (SQ) in the empty group begins storing instructions (i.e., the queue state changes from empty to non-empty), the group's participation status will be restored, and it will rejoin the arbitration process. This means that the participation status of a queue group is dynamic, capable of quickly responding to changes in queue state, ensuring that queues with pending instructions can receive service in a timely manner.

[0062] Optionally, the host sends I / O commands to the NVMe device via the PCIe bus. The NVMe device's storage controller stores these commands in a commit queue (SQ) pre-assigned to that namespace, based on the namespace identifier from which they originate. The NVMe device's firmware divides the SQs into different commit queue groups based on the namespace's service type and service level, and further divides the queue groups into queue sets based on storage media type (e.g., SSD). The firmware assigns a first priority to each queue set according to preset priority rules for different storage media types. The firmware calculates a second priority for each commit queue group, which is equal to the highest priority namespace in its associated namespace set; that is, the firmware traverses all namespaces in each group and selects the highest priority as the group's second priority. At the start of a preset arbitration cycle, the arbitration logic module first selects the queue set to be processed in the current cycle based on the first priority of the queue set. Then, from the selected queue set, it selects the queue group to be processed in the current cycle based on the second priority of the commit queue group. Finally, within the selected queue group, the arbitration module chooses the specific commit queue (SQ) to be processed in the current cycle based on the queue's own priority and scheduling policy (such as WRR, RR, etc.). Instructions in the selected commit queue are passed to the NVMe device's processor for data read / write, status update, and other operations. The processor then feeds back the processing results to the host and prepares for the next arbitration cycle.

[0063] In this embodiment, instructions sent by the host are stored in multiple commit queues, which are pre-divided into multiple commit queue groups. These commit queue groups are then pre-divided into multiple queue sets, constructing a three-level logical grouping structure: "Queue Set (Set) → Commit Queue Group (Group) → Commit Queue (SQ)". By combining storage medium type, service type, and service level, priorities and weights are assigned to different levels, solving the problem of inaccurate priority settings caused by coarse prioritization in related technologies. Through a multi-level nested priority system, precise control is ensured over resource allocation at both the basic physical layer (storage medium) and the logical layer (service type), meeting the stringent QoS requirements of different service and load scenarios. Furthermore, the queue sets are pre-divided according to storage medium type, with different first priorities for different medium types, ensuring that critical services and data access operations can first utilize the most suitable storage resources. The multi-level arbitration mechanism addresses the problem of inefficient resource allocation. Each submission queue group corresponds to multiple namespaces, each with different business types and service levels. The priority of each namespace is determined based on the business type and service level. Each submission queue group also has a secondary priority, determined by the highest priority among the multiple namespaces associated with that group, ensuring priority processing of critical business processes. In the hierarchical arbitration of queue sets, queue groups, and queues, dynamic arbitration is used instead of a single fixed priority or simple round-robin method. This multi-level arbitration approach solves the problems of inaccurate priority settings, inefficient resource allocation, and insufficient fairness in related arbitration mechanisms. By refining the queue management unit, the precision of scheduling is improved, enabling more accurate fulfillment of the Quality of Service (QoS) requirements of various queues in complex multi-tenant and mixed load scenarios, avoiding the "starvation" of low-priority queues. In summary, the multi-level arbitration mechanism, through its flexible hierarchical division and independent scheduling strategy, can more precisely meet the QoS requirements of multi-tenant / multi-business environments, avoiding rigid resource allocation and improving overall service quality and user experience.

[0064] In one exemplary embodiment, before storing the multiple instructions sent by the host into multiple submission queues, the method further includes:

[0065] The length of the preset arbitration period is determined based on the preset input / output operation rate; the preset input / output operation rate is negatively correlated with the length of the preset arbitration period.

[0066] The preset input / output operation rate refers to the target number of input / output operations per second (IOPS) set during the design of the NVMe storage device. This embodiment proposes a preset arbitration period adaptation method to dynamically adjust the length of the preset arbitration period based on the total number of submission queues. It also divides the number of queues for arbitration equally according to the number of CPUs, achieving fast arbitration based on the differences in parameters between different numbers of NVMe queues / groups using grouped parallel processing. A distributed parallel processing strategy is adopted to achieve flexible arbitration configuration in the firmware. In other words, the length of the preset arbitration period is adjusted according to the system's preset IOPS target. The preset input / output operation rate is negatively correlated with the length of the preset arbitration period, meaning that the higher the system's required IOPS, the shorter the arbitration period must be to meet the higher input / output operation rate. This is because a shorter arbitration period allows for more frequent queue scheduling, enabling the device to respond to and process I / O requests faster, ultimately achieving higher IOPS performance. For example, if the system needs to support 4M IOPS, the arbitration period must be ≤1 / 4M seconds (i.e., 250 nanoseconds) to ensure that 4 million I / O operations can be processed per second. Conversely, if the system requires a lower IOPS, the arbitration period can be extended appropriately, because a lower IOPS target allows more time for arbitration decisions without worrying about affecting the overall response speed or processing capacity.

[0067] In some embodiments, this embodiment also provides a strategy for dynamically adjusting the arbitration cycle based on the processor (such as CPU) frequency of the storage device. That is, at the firmware implementation level, the length of the preset arbitration cycle can be finely adjusted according to the actual processor (such as CPU) frequency of the storage device in operation. This means that the firmware calculates the number of arbitrations that can be performed within each preset arbitration cycle based on the CPU's operating frequency, ensuring that the CPU completes at least one queue arbitration process within each arbitration cycle, thus satisfying the IOPS target while adapting to the specific performance of the CPU.

[0068] In this embodiment, the length of the preset arbitration period is dynamically adjusted according to the preset input / output operation rate, and the length of the preset arbitration period is negatively correlated with the preset input / output operation rate. When the system needs to achieve a higher IOPS target, the arbitration period will be set to be shorter to ensure that I / O requests can be responded to and processed quickly, improve scheduling efficiency, and solve the problem of insufficient high-concurrency efficiency in related technologies.

[0069] In one exemplary embodiment, each of the multiple submission queues has a configuration enabled state and a skip flag; the configuration enabled state represents the working state of the corresponding submission queue; the skip flag is used to mark whether the corresponding submission queue participates in arbitration.

[0070] The enabled status is a binary flag indicating whether the commit queue (SQ) in the NVMe storage device is activated and ready to receive and process I / O commands sent by the host. If the queue's enabled status is true (i.e., the queue is enabled), then the queue will be considered a candidate queue in the arbitration process, participating in priority and weight comparisons, and thus having a chance to be scheduled for execution. In other words, a queue with an enabled status indicates that the queue was created and normally enabled by the kernel device driver, and its enabled status is enabled. If the queue's enabled status is false (i.e., the queue is disabled), then the queue will not be considered a candidate queue in the arbitration process, and it will be directly excluded from the scheduling of the current arbitration cycle.

[0071] The Skip Flag (SKIP) is another key identifier used to indicate whether a commit queue in an NVMe storage device should be excluded from the current arbitration cycle, i.e., whether the queue should participate in the arbitration decision. If the queue's Skip Flag is set to true (i.e., skipped), then regardless of its priority or weight settings, the queue will not be included in the candidate queue set for the current arbitration cycle, thus ensuring the accuracy and efficiency of the arbitration process.

[0072] In some embodiments, before selecting the current queue set from multiple queue sets, the method further includes:

[0073] Remove instruction submission queues from multiple submission queues that meet at least one of the following conditions: the corresponding configuration enable state is disabled, the corresponding skip flag indicates that it does not participate in arbitration, and the corresponding output queue does not have any available handles; wherein, the output queue is used to store instructions in the instruction submission queue corresponding to the output queue.

[0074] Specifically, the configuration of a submission queue as enabled or disabled indicates that the queue is disabled and will not be considered a candidate queue for the arbitration process; it will be directly excluded from the scheduling of the current arbitration cycle. The skip flag of a submission queue indicates that it will not participate in arbitration. If the skip flag is set to true (i.e., skipped), the queue will not be included in the candidate queue set for the current arbitration cycle. In other words, the queues retained should be non-SKIP queues.

[0075] The Outbound Queue (OBQ) is a component in NVMe storage devices used to cache and transfer I / O instructions selected from the winning instruction commit queue (SQ) during the arbitration process for processing by the corresponding CPU core.

[0076] A handle is an identifier used to reference or access an object (such as a file, window, memory block, queue, etc.) within the operating system. Handles are typically allocated by the operating system and are opaque pointers or integers. Applications manipulate or access the object through the handle without needing to know the object's specific implementation or storage location. An available handle (HandleCnt) refers to a usable context used to store commands retrieved from the host. Specifically, an available handle refers to a handle in the NVMe storage device's Output Queue (OBQ) that is currently available for caching and processing new I / O instruction data. When an OBQ has available handles (i.e., HandleCnt > 0, indicating at least one free context slot in the OBQ), it means it has sufficient resources to receive instructions from the queue selected during the arbitration process, thus enabling timely processing of these instructions and ensuring efficient queue scheduling and system responsiveness.

[0077] In this embodiment, before selecting the current queue set from multiple queue sets, removing queues whose corresponding OBQs lack available handles avoids resource contention, ensures that the queue selected through arbitration can immediately perform I / O operations, avoids invalid scheduling and resource waiting, and improves resource utilization. Removing queues whose configuration enable state is disabled avoids scheduling queues that are faulty or under maintenance, which helps prevent the propagation of system faults and maintains stable system operation. Removing queues marked with skip flags that do not participate in arbitration can significantly reduce the size of the queue set participating in arbitration, thereby speeding up the arbitration process, reducing arbitration latency, and improving the overall response speed of the storage system.

[0078] In one exemplary embodiment, selecting the current queue set from the multiple queue sets according to a first priority corresponding to the queue set, and selecting the current submission queue group from the current queue set according to a second priority corresponding to the submission queue group from the multiple submission queue groups, includes:

[0079] The set of queues with the highest first priority among multiple queue sets is determined as the current queue set; the group of submission queues with the highest second priority among the current queue sets is determined as the current submission queue group.

[0080] In some embodiments, each commit queue may also have a third priority. In the process of selecting the current commit queue from the current commit queue group, the commit queue with the highest third priority in the current commit queue group may be determined as the current commit queue.

[0081] Optionally, the queue with the highest priority is selected by comparing the Set->Group->Queue levels in sequence. Specifically, in the Set level, the first priority of the Set is compared (e.g., the smaller the value, the higher the priority), and the set of queues with the highest first priority is selected as the current queue set; in the Group level, the second priority of the Group under the current queue set is compared (e.g., the smaller the value, the higher the priority), and the commit queue group with the highest second priority is selected as the current commit queue group; in the Queue level, the third priority of the commit queues under the current commit queue group is compared, and the commit queue with the highest third priority is selected as the current commit queue.

[0082] By consistently selecting the highest priority queue set and queue group in this embodiment, I / O requests for critical operations (such as system configuration updates and fault recovery) can be responded to and processed at the fastest speed, thereby significantly improving the storage device's response speed to emergency events.

[0083] In one exemplary embodiment, the above method further includes:

[0084] When multiple queue sets include multiple queue sets with the highest first priority, the current queue set is selected from these multiple queue sets with the highest first priority through a fair round-robin process; when the current queue set includes multiple submission queue groups with the highest second priority, the current submission queue group is selected from these multiple submission queue groups with the highest second priority through a fair round-robin process.

[0085] In the process of comparing the Set->Group->Queue levels, there may be scenarios where the priorities are the same. In such cases, to ensure that queues, groups, or sets with the same priority can be scheduled in a predetermined order, a fair round-robin approach can be used to select the final winner.

[0086] Fair Round Robin is a scheduling algorithm designed to ensure that resources (such as processor time, access to storage devices, etc.) are fairly allocated among multiple requesters (such as queue sets or queue groups). When multiple queue sets or submission queue groups have the same highest priority, Fair Round Robin selects the next scheduling object through sequential round-robin (i.e., in a fixed order). This dynamic polling order achieves fair scheduling, avoiding resource monopoly problems caused by fixed priorities and preventing some queues from being neglected or "starved" due to a fixed priority allocation mechanism.

[0087] The core of fair round-robin is maintaining a polling pointer that indicates the queue set or group that was previously scheduled. Each time a scheduling occurs, the pointer moves forward to point to the next queue set or group, ensuring that all sets or groups with the same priority have a chance to be scheduled in each polling cycle. For example, suppose an NVMe storage device has the following queue configuration: Queue Sets (Sets): There are three Sets (S1, S2, S3), all with the same first priority (e.g., PRIORITY=1). In the current queue set S1, there are four Groups (G1, G2, G3, G4), all with the same second priority (e.g., PRIORITY=2). The polling pointer initially points to queue set S1. During an arbitration cycle, when multiple queue sets with the highest first priority are encountered, fair round-robin selects the next queue set to schedule in the order S1→S2→S3. If S1 is selected for scheduling in the current polling cycle and the pointer moves to S2, then in the next arbitration cycle, even if S1 has the highest priority again, S2 will be scheduled first until all Sets are scheduled. Only then will the polling pointer return to S1 and a new polling cycle begin.

[0088] In some embodiments, in addition to selecting the highest priority queue set and submission queue group as described in the above embodiments, a fair round-robin method can also be used to select the current queue set and the current submission queue group. That is, the current queue set is selected from multiple queue sets, and the current submission queue group is selected from the current queue set, including: filtering the current queue set from multiple queue sets through a fair round-robin method; and selecting the current submission queue group from the current queue set through a fair round-robin method.

[0089] The fair round-robin method has been explained in the above embodiments and will not be repeated here. Using fair round-robin to select the current queue set and the current submission queue group ensures that each queue set and each queue group has a chance to be scheduled, preventing any group from being ignored for a long time. This achieves even resource allocation and improves overall system performance.

[0090] This embodiment implements fair round-robin among queue sets or groups of equal priority, ensuring that resources are not monopolized by a particular set or group for a long period, but are fairly shared among units of equal priority. This avoids the "starvation" phenomenon caused by uneven resource allocation and simplifies the selection decision among multiple highest priority queue sets or groups. No additional complex algorithms or calculations are needed to determine which set or group has higher priority, thereby reducing the complexity and latency of scheduling decisions, speeding up the scheduling process, and improving the overall response speed of the system.

[0091] In one exemplary embodiment, when the number of submission queues exceeds a thousand (e.g., 1025 SQs), the latency of the related serial arbitration method increases significantly, making it difficult to meet the arbitration cycle requirements of preset processing rates (e.g., 4M / s). Therefore, to solve this problem, in this embodiment, all submission queues are evenly distributed according to the number of processor cores in the storage device, forming multiple groups (i.e., multiple groups of submission queues). Each group has the same number of submission queues to achieve parallel arbitration and processing. Assuming the storage device has 1025 SQ queues and the processor of the storage device has 4 cores, each core corresponds to 256 SQ queues.

[0092] Multiple submission queues correspond one-to-one with multiple cores of the processor in the storage device. Each group (i.e., each submission queue) is assigned to a specific core of the processor for arbitration and processing, thereby improving concurrent processing capabilities. It can be understood that this embodiment utilizes multiple cores in the storage device to independently arbitrate their corresponding queue sets, queue groups, and queues within multiple preset arbitration cycles, achieving parallel arbitration processing. This significantly shortens the arbitration cycle, improves system response speed and concurrent processing capabilities, and solves the efficiency bottleneck faced by serial arbitration methods in high-concurrency scenarios in related technologies.

[0093] Processors in storage devices typically contain multiple cores. Each processor core is part of the storage device's internal processing unit and can execute instructions independently. In this embodiment, multiple cores process their respective queues in parallel, enhancing the overall performance of the device. Each core can process instructions from multiple corresponding commit queues (SQs) in parallel. This design helps improve the storage device's concurrent processing capabilities and overall performance. By assigning different SQs to different cores, or having each core manage a set of SQs, instruction processing latency can be effectively reduced, resource utilization optimized, and I / O operation response speed accelerated, especially in data center environments handling a large number of concurrent requests.

[0094] In some embodiments, processing instructions stored in the current commit queue of the current commit queue group by the processor of the storage device includes:

[0095] The current commit queue is selected sequentially from the current commit queue group through fair polling. The instructions stored in the selected current commit queue are stored in the output queue group. The instructions stored in the output queue group are processed in parallel by the designated core corresponding to the current commit queue. The output queue group corresponds to the designated core. The output queue group includes multiple output queues. The multiple output queues are used to store the instructions stored in the selected current commit queue.

[0096] The current submission queue can be selected using either a priority-based selection method or a fair round-robin method. In this embodiment, a fair round-robin method is used to select the current submission queue. As can be seen from the above embodiment, the current queue set and current submission queue group can be selected using a priority-based selection method, while the current submission queue can be selected using a fair round-robin method.

[0097] In this context, an output queue group refers to a collection of queues within the NVMe storage device used for further processing and sending I / O instructions to the physical storage medium. An output queue group comprises multiple output queues, each designed to store instructions scheduled from the currently committed queue. In this embodiment, the output queue group is designed to correspond to a specific core, allowing each core to independently process instructions in its output queue, improving parallel processing efficiency. It can be understood that, to achieve high concurrency processing, each output queue group is assigned to an independent core, avoiding resource contention between cores and accelerating instruction processing speed.

[0098] In this embodiment, by distributing instructions to multiple output queues and having them processed in parallel by designated CPU cores, the parallel processing capability of the system is greatly enhanced, the instruction processing wait time is reduced, and the overall I / O performance and response speed are improved. The correspondence between the output queue groups and the designated cores ensures that each core is responsible for processing one set of output queues. This not only reduces resource contention between cores but also improves the efficiency of resource allocation, enabling each core to make efficient use of its processing power.

[0099] In one exemplary embodiment, fair polling includes the following steps:

[0100] 1. During non-initial scheduling processes, determine the first offset of the object identifier of the current object to be compared relative to the object identifier of the object to be compared pointed to by the polling pointer, and determine the second offset of the object identifier of the current temporary winning object relative to the object identifier of the object to be compared pointed to by the polling pointer; the current object to be compared is one of the following: queue set, submission queue group, submission queue; the polling pointer is used to point to the object to be compared polled in the previous scheduling process of the current scheduling process; the current temporary winning object refers to the target object selected in the previous scheduling process;

[0101] 2. If the first offset is less than the second offset, the polling order of the current object to be compared is determined to be higher than that of the currently temporarily stored winning object. The currently temporarily stored winning object is set as the current object to be compared, the polling pointer is set to point to the current object to be compared, and the current object to be compared is determined as the target object selected by the current scheduling process. The target object includes one of the following: the current queue set, the current submission queue group, and the selected current submission queue.

[0102] The scheduling process refers to the process of selecting a target object from a set of queues, a group of submission queues, or a submission queue for instruction processing. Each cycle's scheduling process is independent, but continuity and fairness are maintained through a polling pointer and offset mechanism. The target object is the object selected through a fair polling process; it can be the current set of queues, the current group of submission queues, or the selected current submission queue, and is the direct output of the scheduling process.

[0103] The object to be compared refers to the object whose scheduling priority the system is evaluating during the scheduling cycle. It can be a queue set, a submission queue group, or a submission queue, depending on the level of the scheduling process. The object identifier of the object to be compared is an identifier used to uniquely identify each object to be compared, such as the ID of the queue set, submission queue group, or submission queue.

[0104] In the process of fair polling, the polling pointer is used to record the winning object in the previous scheduling process, serving as the starting point of the current scheduling cycle to achieve cyclic scheduling and ensure fairness.

[0105] The currently stored winning object refers to the target object selected in the previous scheduling process during the non-first scheduling process. It is used to compare the offset with the currently compared object to determine the scheduling order.

[0106] The first offset refers to the offset of the object identifier of the currently compared object relative to the object identifier pointed to by the polling pointer in the previous scheduling process during non-initial scheduling. It is used to determine the scheduling order. Specifically, the first offset is determined by calculating the numerical difference between the object identifier of the currently compared object and the object identifier pointed to by the polling pointer, and then taking the modulo of the total number of objects. The first offset is used to evaluate the order of the currently compared object in the polling sequence. A smaller first offset means that the currently compared object is closer to the object pointed to by the polling pointer after the previous scheduling, and therefore has a higher scheduling priority in the current scheduling.

[0107] Similarly, the second offset refers to the offset of the object identifier of the currently temporarily winning object relative to the object identifier pointed to by the polling pointer in the previous scheduling process, also in non-initial scheduling. It is used for comparing the scheduling order. Similar to the first offset, the second offset is also determined by the numerical difference and modulo operation between object identifiers, reflecting the position information of the temporarily winning object in the polling sequence. By comparing the first and second offsets, the system can determine whether the object to be compared should be selected as the target object of the current scheduling process before the temporarily winning object.

[0108] When the first offset is less than the second offset, it indicates that the current object to be compared is further back in the polling sequence relative to the polling pointer, meaning it has been waiting longer since the last scheduling. Based on the principle of fairness, the system will prioritize scheduling objects with longer waiting times, thus setting the currently temporarily stored winning object as the current object to be compared, updating the polling pointer to point to the current object to be compared, and determining the current object to be compared as the target object of the current scheduling process. This ensures that all queues or queue groups are treated fairly during scheduling, avoiding long-term imbalances in resource allocation and the problem of a particular queue being "starved."

[0109] In one exemplary embodiment, the above method further includes:

[0110] During the initial scheduling process, the polling pointer is set to invalid, and the object with the smaller object identifier between the currently temporary winning object and the currently uncompared object is set as the currently temporary winning object. The polling pointer is then set to the currently uncompared object.

[0111] In this embodiment, when the storage device starts scheduling the queue, it initializes the scheduling process by comparing simple numerical values, selecting the object with the smallest identifier as the winner of the first scheduling, and updating the polling pointer to this object. This provides a clear starting point for subsequent scheduling cycles, ensuring the continuity and fairness of the scheduling process. This mechanism avoids the uncertainty that may occur during the first scheduling and lays the foundation for a subsequent offset-based fair polling strategy.

[0112] In some embodiments, the fair round-robin method can employ the RR_Select() round-robin scheduling mechanism, wherein the input parameters of the RR_Select() round-robin scheduling mechanism include the following:

[0113] total: The total number of objects to be compared (e.g., the total number of queues, the total number of groups, the total number of sets).

[0114] prev_id: The ID of the object to be compared in the previous scheduling process of the current scheduling process (polling pointer, recording the historical scheduling position);

[0115] curr_id: The ID of the object to be compared;

[0116] winner_id: The ID of the currently stored winning object.

[0117] The output parameters of the RR_Select() round-robin scheduling mechanism include the following:

[0118] TRUE: curr_id should take precedence over winner_id in the polling order;

[0119] FALSE: curr_id should not take precedence over winner_id.

[0120] The pseudocode implementation logic of the RR_Select() round-robin scheduling mechanism is as follows:

[0121] def RR_ Select (total, prev_id, curr_id, winner_id):

[0122] # If prev_id is invalid (e.g., during the first scheduling), directly compare the numerical values.

[0123] if prev_id >= total:

[0124] return curr_id < winner_id

[0125] # Calculate the order of curr_id and winner_id relative to prev_id

[0126] curr_offset = (curr_id - prev_id) % total

[0127] winner_offset = (winner_id - prev_id) % total

[0128] # If the offset of curr_id is smaller, it means that it is earlier in the polling order.

[0129] return curr_offset < winner_offset

[0130] Here, curr_offset represents the first offset; winner_offset represents the second offset.

[0131] For example, suppose the objects to be compared are commit queues, and the NVMe storage device includes three commit queues: Queue1, Queue2, and Queue3. During the initial scheduling process, the polling pointer is initialized to invalid, i.e., the polling pointer `prev_id` is set to an invalid value, such as -1. Assuming the system starts its initial scheduling, it compares the object identifiers of Queue1 and Queue2. The object identifiers of Queue1 and Queue2 are 1 and 2 respectively. The `winner_id` (the currently temporary winning object) will be set to the object with the smaller ID, therefore, `winner_id` = 1. Queue2 becomes the current object to be compared, so `curr_id` (the ID of the current object to be compared) is set to 2, i.e., `curr_id` = 2. Since it is the first scheduling, the polling pointer `prev_id` is updated to the value of `curr_id`, i.e., `prev_id` = 2. The RR_Select() algorithm outputs FALSE. In the next scheduling process, Queue3 becomes the current object to be compared, i.e., curr_id=3. Based on the position of the current object to be compared, Queue3 (ID 3), relative to the polling pointer prev_id (currently 2), we calculate (3-2)%N=1 (where N is the total number of objects, assumed to be 3, because we only consider the case of 3 queues). Here, (3-2)%3=1 indicates that Queue3 is directly after Queue2. Therefore, the first offset curr_offset is 1. Based on the position of the currently temporarily stored winning object, Queue1 (ID 1), relative to the polling pointer prev_id, we calculate (1-2)%N=2. Here, the result of calculating (1-2) is -1, and the calculation of -1%3 needs to be interpreted. -1 can be understood as the offset from Queue2 to Queue1. (1-2)%3=2 means that Queue1 has an offset of 2 positions relative to Queue2 in the round-robin scheduling, that is, it returns to Queue1 through Queue3. Therefore, winner_offset is 2. Since curr_offset is less than winner_offset, the output of the RR_Select() algorithm is TRUE. At this point, Queue3 (curr_id is 3) becomes the new winner, winner_id is set to 3, and prev_id is updated to the ID of Queue3, which is 3. Based on the judgment result of RR_Select(), Queue3 becomes the target object of the current scheduling process, which means that its internal I / O instructions will be preferentially transmitted to the associated output queue group, and then processed by the corresponding core.

[0132] In this embodiment, the polling pointer is initialized to invalid during the first scheduling, and the scheduling order of the initial queue is determined based on the size of the object identifier. The first currently temporarily stored winning object can be determined by a simple comparison of object identifiers, which improves the initial startup efficiency of the arbitration mechanism and simplifies the initial configuration. In non-first scheduling, the concepts of first offset and second offset are introduced to determine the scheduling priority, which ensures the fairness of the queue in scheduling and can effectively avoid the situation where low-priority queues are ignored or "starved" for a long time, even in high-concurrency scenarios.

[0133] In one exemplary embodiment, the object to be compared currently has a preset weight. The preset weight refers to the initial weight value assigned to the object to be compared, used for queue scheduling decisions in a multi-level priority arbitration mechanism. The higher the preset weight of the object to be compared, the higher the probability that the object will be selected in the scheduling process.

[0134] In some embodiments, after determining the current object to be compared as the target object selected by the current scheduling process, the above method further includes:

[0135] The latest weight of the object to be compared is reduced by a fixed step size to obtain the updated latest weight. The latest weight refers to the weight value of the object to be compared after it was identified as the target object in the historical scheduling process and updated by a fixed step size. If the updated latest weight is 0, the latest weight of the object to be compared is reset to the preset weight, and the polling pointer is updated to point to the object to be compared.

[0136] In this context, the latest weight refers to the updated value of the currently compared object after it won the most recent arbitration, after its weight has been reduced (e.g., by a fixed step size). The latest weight reflects the dynamic priority of the queue and its resource usage in the most recent scheduling cycle. Each time the currently compared object is selected as the winner, its preset weight is reduced by a preset fixed step size to obtain the updated latest weight, thereby achieving dynamic weight adjustment and maintaining scheduling fairness.

[0137] During the process of updating the weight of the current comparison object, a weight counter (WCNT) is used for statistics and updates. Specifically, the weight calculator for the current comparison object is initialized, i.e., WCNT=WEIGHT (i.e., the preset weight). Each time the current comparison object is selected, the count value of the weight calculator (i.e., the latest weight of the current comparison object) is decreased by a fixed step to obtain the updated weight. When the updated weight of the current comparison object drops to 0, i.e., WCNT=0, it indicates that the resource allocation of the current comparison object has reached its minimum allocation limit. At this time, by resetting the latest weight of the current comparison object to the preset weight, it is ensured that the current comparison object can continue to participate in scheduling, restoring its scheduling opportunity and avoiding permanent resource starvation. When the updated weight is 0, the polling pointer is updated to point to the current comparison object, ensuring that the next scheduling starts from it, achieving fair cyclic scheduling.

[0138] As can be seen from the above embodiments, the object to be compared can be any of the following: a set of queues, a group of submission queues, and a queue of submissions. Therefore, in this embodiment, the polling pointer can be the polling pointer of the set of queues, i.e., s_winner_SPREV; the polling pointer of the group of submission queues, i.e., g_winner_GPREV; or the polling pointer of the queue of submissions, i.e., q_winner_QPREV. The type of the polling pointer depends on the type of the object to be compared. For example, the global variables corresponding to the set of queues, the group of submission queues, and the queue of submissions are initialized respectively. The global variables corresponding to the set of queues include s_winner (the currently polled value of the set) and s_winner_SPREV; the global variables corresponding to the group of submission queues include g_winner (the currently polled value of the group) and g_winner_GPREV; and the global variables corresponding to the queue of submissions include q_winner (the currently polled value of the queue) and q_winner_QPREV. For a submission queue group, its polling pointer is g_winner_GPREV. If the weight count (WCNT) of a group decreases to 0, it is reset to the group's preset weight WEIGHT, and the group's polling pointer g_winner_GPREV is updated to point to the currently compared object. Similarly, when the weight count (WCNT) of a set decreases to 0, the set's polling pointer s_winner_SPREV is reset and updated.

[0139] For example, an NVMe storage device has three commit queues, Queue1, Queue2, and Queue3, with IDs 1, 2, and 3 respectively. Assume that in a series of scheduling operations, the default order of the winning objects is as follows:

[0140] First scheduling: Queue1 becomes the winner, and the latest weights of Queue1, Queue2, and Queue3 are all the preset weights WEIGHT.

[0141] Second scheduling: Queue2 becomes the winner, the latest weight of Queue1 is reduced by a fixed step size STEP, and the updated latest weight is WEIGHT-STEP. The latest weights of Queue2 and Queue3 are both the preset WEIGHT.

[0142] Third scheduling: Queue3 becomes the winner, the latest weight of Queue2 is reduced, the updated latest weight is WEIGHT-STEP, and the latest weight of Queue3 is WEIGHT because it won for the first time.

[0143] Fourth scheduling: Queue1 becomes the winner, the latest weight of Queue3 is reduced, and the updated latest weight is WEIGHT-STEP. The latest weight of Queue1 is reduced again, and the updated latest weight is WEIGHT-2×STEP.

[0144] Suppose that during N scheduling iterations, Queue1 wins, and its latest weight has dropped to 0 due to multiple scheduling attempts. At this point, Queue1's latest weight is reset to the preset weight WEIGHT. In subsequent scheduling iterations, Queue1 regains its competitive position with Queue2 and Queue3.

[0145] Figure 4 This is an architecture diagram of a queue grouping architecture according to an embodiment of this application, as shown below. Figure 4As shown, multiple submission queues (submission queues 1-N) are divided into a three-layer structure (Set→Group→Queue). In the Set layer, the WRR (SP) algorithm is used to select the current queue set. The WRR (SP) algorithm refers to a specific processing weighted round-robin algorithm (or single-point weighted round-robin algorithm). Based on the weighted round-robin algorithm, it performs weighted scheduling on multiple queue sets, prioritizing certain queue sets according to pre-set weights to ensure that queues or tasks with higher weights receive more processing opportunities. In the case of queues with the same priority, the RR (Set) algorithm is used to round-robin schedule multiple queue sets, ensuring that queues in each set have a fair processing opportunity. The WRR (SP) algorithm is used to process multiple submission queue groups in the current queue set to obtain the current submission queue group. In the case of the same priority, the RR (Set) algorithm is used to poll and schedule multiple submission queue groups in the current queue set to obtain the current submission queue group. The RR (Set) algorithm is then used to poll the submission queues in the current submission queue group to obtain the current submission queue. The instructions in the current submission queue are stored in the output queue group, and the processor processes the instructions in the output queue group.

[0146] Optionally, this embodiment performs arbitration processing on multiple NVMe submission queues by setting an initialization module, a candidate filtering module, a multi-level arbitration module, and a winning processing status update module, thereby realizing multi-level priority nested scheduling (Set→Group→Queue), satisfying multi-tenant QoS, ensuring the fairness of queue processing under different weights and the same priority, and adopting sequential round-robin scheduling of queues to avoid the problem of host IO timeout caused by a certain queue being starved.

[0147] In this embodiment, the preset weight of the current comparison object is dynamically adjusted and combined with polling. The higher the preset weight, the more arbitration cycles are obtained. At the same time, fairness is guaranteed under the same priority. The sequential cyclic scheduling of the queue polling is used to avoid a queue being starved and to ensure scheduling fairness.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0149] Embodiments of this application also provide a submission queue arbitration device for a storage device. Figure 5 This is a structural block diagram of a submission queue arbitration of a storage device according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0150] The queue storage module 502 is used to store multiple instructions sent by the host into multiple submission queues. These submission queues are pre-divided into multiple submission queue groups; these groups are further pre-divided into multiple queue sets; each queue set has a first priority; the queue sets are divided according to storage medium type; different storage medium types correspond to different priorities; the first priority of each queue set is determined based on the priority corresponding to its storage medium type; each submission queue group has a second priority; each submission queue group corresponds to multiple namespaces; each namespace has a different priority; each namespace corresponds to different service types and service levels; the priority of each namespace is determined based on the service type and service level; the second priority of each submission queue group is the highest priority among the multiple namespaces corresponding to that group.

[0151] The queue arbitration module 504 is used to execute the following arbitration process in multiple preset arbitration cycles: select the current queue set from multiple queue sets according to the first priority corresponding to the queue set in multiple queue sets; select the current submission queue group from the current queue set according to the second priority corresponding to the submission queue group in multiple submission queue groups; and process the instructions stored in the current submission queue in the current submission queue group through the processor of the storage device.

[0152] The above-described device stores host-sent instructions in multiple submission queues. These queues are pre-divided into multiple submission queue groups, which are then further divided into multiple queue sets. Dynamic arbitration, rather than a single fixed priority or simple round-robin method, is used in the hierarchical arbitration of queue sets, queue groups, and queues. This multi-level queue arbitration method solves the problems of inaccurate priority settings, inefficient resource allocation, and insufficient fairness in related arbitration mechanisms. By refining the queue management unit, scheduling accuracy is improved, enabling more precise fulfillment of the Quality of Service (QoS) requirements of various queues in complex multi-tenant and mixed load scenarios, avoiding the "starvation" of low-priority queues. In summary, the multi-level arbitration mechanism, through its flexible hierarchical division and independent scheduling strategy, can more precisely meet the QoS requirements of multi-tenant / multi-service scenarios, avoid rigid resource allocation, and improve overall service quality and user experience.

[0153] Optionally, the queue arbitration module 504 is also used to determine the length of the preset arbitration period based on the preset input / output operation rate; the preset input / output operation rate is negatively correlated with the length of the preset arbitration period.

[0154] Optionally, each of the multiple submission queues has a configuration enabled state and a skip flag; the configuration enabled state indicates the working state of the corresponding submission queue; the skip flag is used to mark whether the corresponding submission queue participates in arbitration; before selecting the current queue set from the multiple queue sets, the queue arbitration module 504 is also used to remove instruction submission queues from the multiple submission queues that meet at least one of the following conditions: the corresponding configuration enabled state is disabled, the corresponding skip flag indicates that it does not participate in arbitration, and there is no available handle in the corresponding output queue; wherein, the output queue is used to store the instructions in the instruction submission queue corresponding to the output queue.

[0155] Optionally, the queue arbitration module 504 is also used to determine the queue set with the highest first priority among multiple queue sets as the current queue set; and to determine the submission queue group with the highest second priority among the current queue sets as the current submission queue group.

[0156] Optionally, the queue arbitration module 504 is further configured to select the current queue set from the multiple queue sets with the highest first priority through fair round-robin when the multiple queue sets include multiple queue sets with the highest first priority; and to select the current submission queue group from the multiple submission queue groups with the highest second priority through fair round-robin when the current queue set includes multiple submission queue groups with the highest second priority.

[0157] Optionally, the processor includes multiple cores; multiple submission queues are pre-divided into multiple groups of submission queues according to the number of processor cores, and the multiple groups of submission queues correspond one-to-one with the multiple cores of the processor; the queue arbitration module 504 is also used to select the current submission queue sequentially from the current submission queue group in a fair round-robin manner, store the instructions stored in the selected current submission queue into the output queue group, and process the instructions stored in the output queue group in parallel by the designated core corresponding to the current submission queue; the output queue group corresponds to the designated core; the output queue group includes multiple output queues; the multiple output queues are used to store the instructions stored in the selected current submission queue.

[0158] Optionally, the queue arbitration module 504 is further configured, during non-initial scheduling processes, to determine, in the following ways, a first offset of the object identifier of the current object to be compared relative to the object identifier of the object to be compared pointed to by the polling pointer, and a second offset of the object identifier of the current temporarily stored winning object relative to the object identifier of the object to be compared pointed to by the polling pointer; the current object to be compared is one of the following: a queue set, a submission queue group, or a submission queue; the polling pointer is used to point to the object to be compared polled in the previous scheduling process of the current scheduling process; the current temporarily stored winning object refers to the target object selected in the previous scheduling process; if the first offset is less than the second offset, the polling order of the current object to be compared is determined to be higher than that of the current temporarily stored winning object, the current temporarily stored winning object is set as the current object to be compared, the polling pointer is set to point to the current object to be compared, and the current object to be compared is determined as the target object selected in the current scheduling process; the target object includes one of the following: the current queue set, the current submission queue group, or the selected current submission queue.

[0159] Optionally, the queue arbitration module 504 is also used to, during the initial scheduling process, set the polling pointer to invalid, set the object with the smaller object identifier between the currently temporarily stored winning object and the currently compared object as the currently temporarily stored winning object, and point the polling pointer to the currently compared object.

[0160] Optionally, the current object to be compared has a preset weight. After the current object to be compared is determined to be the target object selected in the current scheduling process, the queue arbitration module 504 is also used to reduce the latest weight of the current object to be compared by a fixed step size to obtain the updated latest weight. The latest weight refers to the weight value of the current object to be compared after it was determined to be the target object in the historical scheduling process and updated by a fixed step size. If the updated latest weight is 0, the latest weight of the current object to be compared is reset to the preset weight, and the polling pointer is updated to point to the current object to be compared.

[0161] For a description of the features of the submission queue arbitration device in the embodiment of the storage device, please refer to the relevant description of the submission queue arbitration method in the embodiment of the storage device, which will not be repeated here.

[0162] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in the submission queue arbitration method embodiments of any of the above-described storage devices.

[0163] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the submission queue arbitration method embodiments of any of the above storage devices when running.

[0164] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0165] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the embodiments of the submission queue arbitration method for any of the above-described storage devices.

[0166] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the submission queue arbitration method embodiment of any of the above storage devices.

[0167] According to another aspect of the embodiments of this application, a computer program product is also provided, comprising a computer program / instructions containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit 601, it performs various functions provided in the embodiments of this application. The sequence numbers of the embodiments of this application above are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0168] Figure 6 A schematic block diagram of a computer system architecture for implementing embodiments of the present application is shown. Figure 6 As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which performs various appropriate actions and processes based on programs stored in ROM 602 or loaded into RAM 603 from storage section 608. Random access memory 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0169] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card, such as a local area network card or modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0170] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit 601, it performs various functions defined in the system of this application.

[0171] It should be noted that, Figure 6 The computer system 600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0172] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0173] The above provides a detailed description of a submission queue arbitration method and electronic device for a storage device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for arbitration of a submission queue of a storage device, the method comprising: The method comprises: storing a plurality of instructions sent by a host into a plurality of submission queues; the plurality of submission queues are divided into a plurality of submission queue groups in advance; the plurality of submission queue groups are divided into a plurality of queue sets in advance; a queue set in the plurality of queue sets corresponds to a first priority; the queue sets in the plurality of queue sets are divided according to storage medium types; different storage medium types correspond to different priorities; the first priority corresponding to a queue set in the plurality of queue sets is determined according to the priority corresponding to the storage medium type of the queue set; a submission queue group in the plurality of submission queue groups corresponds to a second priority; a submission queue group in the plurality of submission queue groups corresponds to a plurality of namespaces; the plurality of namespaces correspond to different priorities; a namespace in the plurality of namespaces corresponds to different service types and service levels; the priority corresponding to a namespace in the plurality of namespaces is determined according to the service type and the service level; the second priority corresponding to a submission queue group in the plurality of submission queue groups is the maximum priority in the plurality of namespaces corresponding to the submission queue group in the plurality of submission queue groups; in a plurality of preset arbitration periods, the following arbitration processes are respectively performed: according to the first priority corresponding to a queue set in the plurality of queue sets, a current queue set is selected from the plurality of queue sets; according to the second priority corresponding to a submission queue group in the plurality of submission queue groups, a current submission queue group is selected from the current queue set; and instructions stored in a current submission queue in the current submission queue group are processed by a processor of the storage device.

2. The method of claim 1, wherein, Before the plurality of instructions sent by the host are stored into the plurality of submission queues, the method further comprises: determining the length of the preset arbitration period according to a preset input / output operation rate; the preset input / output operation rate is negatively related to the length of the preset arbitration period.

3. The method of claim 1, wherein, a submission queue in the plurality of submission queues corresponds to a configuration enabled state and a skip flag; the configuration enabled state represents the working state of the corresponding submission queue; the skip flag is used to mark whether the corresponding submission queue participates in arbitration; before the current queue set is selected from the plurality of queue sets, the method further comprises: eliminating an instruction submission queue in the plurality of submission queues that meets at least one of the following conditions: the corresponding configuration enabled state is a disabled state, the corresponding skip flag represents not participating in arbitration, and there is no available handle in the output queue corresponding to the instruction submission queue; wherein the output queue is used to store instructions in the instruction submission queue corresponding to the output queue.

4. The method of claim 1, wherein, the current queue set is selected from the plurality of queue sets according to the first priority corresponding to a queue set in the plurality of queue sets, and the current submission queue group is selected from the current queue set according to the second priority corresponding to a submission queue group in the plurality of submission queue groups, comprising: The queue set with the highest first priority in the plurality of queue sets is determined as the current queue set; and the submission queue group with the highest second priority in the current queue set is determined as the current submission queue group.

5. The method of claim 4, wherein, The method further comprises: In a case where the plurality of queue sets includes a plurality of queue sets with the highest first priority, the current queue set is selected from the plurality of queue sets with the highest first priority by fair polling; In a case where the current queue set includes a plurality of submission queue groups with the highest second priority, the current submission queue group is selected from the plurality of submission queue groups with the highest second priority by the fair polling.

6. The method of claim 1, wherein, The processor includes a plurality of cores; the plurality of submission queues are divided into a plurality of submission queue groups in advance according to the number of cores of the processor, and the plurality of submission queue groups correspond to the plurality of cores of the processor one by one; and the processor of the storage device processes the instructions stored in the current submission queue in the current submission queue group, which comprises: The current submission queue is sequentially selected from the current submission queue group by fair polling, the instructions stored in the selected current submission queue are stored in the output queue group, and the instructions stored in the output queue group are processed in parallel by a specified core corresponding to the current submission queue; the output queue group corresponds to the specified core; the output queue group includes a plurality of output queues; and the plurality of output queues are used to store the instructions stored in the selected current submission queue.

7. The method according to claim 5 or 6, characterized in that, The fair polling mode comprises the following steps: In a non-first scheduling process, a first offset of an object identifier of a current comparison object corresponding to a current scheduling process relative to an object identifier of a comparison object pointed to by a polling pointer is determined, and a second offset of an object identifier of a current temporary winning object corresponding to the current scheduling process relative to the object identifier of the comparison object pointed to by the polling pointer; the current comparison object is one of the following: a queue set, a submission queue group, and a submission queue; the polling pointer is used to point to a comparison object polled in a last scheduling process of the current scheduling process; and the current temporary winning object refers to a target object selected in the last scheduling process; In a case where the first offset is less than the second offset, it is determined that the polling order of the current comparison object is prior to that of the current temporary winning object, the current temporary winning object is set as the current comparison object, the polling pointer is set to point to the current comparison object, and the current comparison object is determined as the target object selected in the current scheduling process; the target object includes one of the following: the current queue set, the current submission queue group, and the selected current submission queue.

8. The method of claim 7, wherein, The method further comprises: In a first scheduling process, the polling pointer is set to be invalid, the object with a smaller object identifier between the current temporary winning object and the current comparison object is set as the current temporary winning object, and the polling pointer is set to point to the current comparison object.

9. The method of claim 7, wherein, The current object to be compared corresponds to a preset weight; After determining the current object to be compared as the target object selected by the current scheduling process, the method further comprises: decreasing the latest weight of the current object to be compared by a fixed step to obtain an updated latest weight; the latest weight refers to the weight value of the current object to be compared after being determined as the target object in the historical scheduling process and updated by the fixed step; In the case that the updated latest weight is 0, resetting the latest weight of the current object to be compared to the preset weight and updating the polling pointer to point to the current object to be compared.

10. An electronic device, comprising: Comprise: a memory for storing a computer program; a processor for executing the computer program to realize the steps of the method according to any one of claims 1 to 9.

Images