Fine-grained qos isolation method and device for nvme-of storage

Abstract

The application discloses a fine-grained QoS isolation method and device for NVMe-oF storage, relates to the technical field of data transmission, and mainly solves the problem of tail delay amplification caused by cross-tenant I / O interference in a multi-tenant environment, and the method comprises the following steps: acquiring system telemetry data and multi-tenant I / O request flow in a multi-tenant scenario; performing micro-slicing processing on the multi-tenant I / O request flow, and importing each micro-slice obtained into a corresponding virtual queue; determining a two-dimensional credit vector for controlling each virtual queue based on the system telemetry data; and scheduling and gating the micro-slices in each virtual queue on a preset time grid based on the two-dimensional credit vector of each virtual queue. The application effectively suppresses micro-bursts in the RNIC near end, cuts off the delay amplification path, reduces I / O tail delay, and improves the service quality predictability in a shared storage environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data transmission, in particular to a fine-grained QoS isolation method and device for NVMe-oF storage. BACKGROUND

[0002] With the evolution of data centers to a storage-compute separation architecture, high-performance NVMe-oF (NVM Express over Fabrics) storage technology based on remote direct memory access (RDMA) has become the core of building an efficient storage resource pool. However, in a multi-tenant shared cloud environment and enterprise data center, the I / O (input / output) streams of different applications or tenants converge on the shared network and storage path, causing serious performance interference problems. Therefore, how to provide predictable, fine-grained quality of service (QoS) isolation for multi-tenants under the NVMe-oF architecture has become a major technical challenge and research significance to release its full performance potential and ensure the stable operation of critical business.

[0003] Currently, there are various technical solutions for QoS management of storage systems. For example, on the host side, the operating system level provides I / O controllers such as cgroups, which can limit the IOPS (input / output per second) or bandwidth of block devices. At the network level, data center networks use mechanisms such as priority-based flow control (PFC) and explicit congestion notification (ECN) to manage congestion and avoid packet loss; some high-end network switches also support scheduling algorithms such as weighted fair queue (WFQ) to provide differentiated services for traffic from different sources. At the storage target side, the NVMe specification itself also defines namespace-level QoS management functions, allowing administrators to set upper limits on IOPS or bandwidth for different namespaces. However, existing solutions still have several deep technical problems when dealing with fine-grained QoS isolation in the NVMe-oF multi-tenant scenario. These problems mainly arise from the coarse scheduling granularity, the single dimension of congestion control, and the lack of end-to-end path coordination, resulting in tail latency amplification. SUMMARY

[0004] Therefore, the present application provides a fine-grained QoS isolation method and device for NVMe-oF storage, mainly to solve the problem of tail latency amplification caused by cross-tenant I / O interference in a multi-tenant environment.

[0005] According to one aspect of the present application, a fine-grained QoS isolation method for NVMe-oF storage is provided, comprising:

[0006] Obtaining system telemetry data and multi-tenant I / O request streams in a multi-tenant scenario;

[0007] micro-slice the multi-tenant I / O request stream and import each micro-slice obtained into a corresponding virtual queue;

[0008] determine a two-dimensional credit vector for controlling each virtual queue based on the system telemetry data;

[0009] schedule and gate the micro-slices in each virtual queue on a preset time grid based on the two-dimensional credit vector of each virtual queue.

[0010] According to another aspect of the present application, there is provided a fine-grained QoS isolation device for NVMe-oF storage, comprising:

[0011] an acquisition module configured to acquire system telemetry data in a multi-tenant scenario and a multi-tenant I / O request stream;

[0012] a micro-slice processing module configured to micro-slice the multi-tenant I / O request stream and import each micro-slice obtained into a corresponding virtual queue;

[0013] a credit evaluation module configured to determine a two-dimensional credit vector for controlling each virtual queue based on the system telemetry data;

[0014] a scheduling module configured to schedule and gate the micro-slices in each virtual queue on a preset time grid based on the two-dimensional credit vector of each virtual queue.

[0015] By means of the above technical solutions, the technical solutions provided by the embodiments of the present application have at least the following advantages:

[0016] The present application provides a fine-grained QoS isolation method and device for NVMe-oF storage. Compared with the prior art, the present application obtains a multi-tenant I / O request stream and micro-slices the multi-tenant I / O request stream to obtain a basic execution unit with finer granularity. The present application acquires system telemetry data in a multi-tenant scenario and determines a two-dimensional credit vector for controlling each virtual queue based on the system telemetry data. The present application schedules and gates the micro-slices in each virtual queue on a preset time grid based on the two-dimensional credit vector of each virtual queue. This can effectively alleviate the RNIC near-end resource competition of a multi-tenant shared NVMe-oF storage system caused by the micro-bursts of different tenant I / O streams, thereby effectively avoiding the problem of tail delay amplification and improving the quality of service and predictability in a shared storage environment.

[0017] The above description is only a summary of the technical solutions of the present application. In order to enable a more thorough understanding of the technical means of the present application, the content of the specification can be implemented, and in order to enable the above and other purposes, features and advantages of the present application to be more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0018] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. Furthermore, the same reference numerals in the drawings indicate the same or similar components. In the drawings:

[0019] Figure 1 A flowchart of a fine-grained QoS isolation method for NVMe-oF storage provided by an embodiment of the present application is shown;

[0020] Figure 2 A flowchart of a method for decomposing a multi-tenant I / O request stream into a predetermined number of micro-slices provided by an embodiment of the present application is shown;

[0021] Figure 3 A flowchart of a method for generating a two-dimensional credit vector provided by an embodiment of the present application is shown;

[0022] Figure 4 A flowchart of a processing method in the case of interruption and preemption provided by an embodiment of the present application is shown;

[0023] Figure 5 A flowchart of a credit transfer method provided by an embodiment of the present application is shown;

[0024] Figure 6 A flowchart of a scheduling and gating release method provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood, and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0026] In the research, it is found that the operation granularity of the existing scheduling mechanism stays at the I / O request or network packet level, at the RNIC (RDMA Network Interface Card) near end of the RDMA network interface card, once a large-size I / O request is submitted to the hardware sending queue, the hardware resource is occupied for a long time, the non-preemptible head blocking is formed for the small but time-delay-sensitive request from other tenants, and the microsecond-level delay peak is caused. In addition, the network-level PFC and other congestion control mechanisms are one-size-fits-all, which cannot distinguish between the instantaneous congestion caused by short-time micro-burst and the continuous congestion caused by long-term rate exceeding, and often pause the whole link due to the burst of a single tenant, causing unnecessary collateral damage to other well-behaved tenants, and even inducing PFC storm. Finally, the QoS strategies of nodes (hosts, networks, and storage) are independent of each other and lack of coordination, the release rhythm of the network side often mismatches the actual service rhythm of the target storage device, causing that even if the traffic is smoothly transmitted in the network, the queuing bottleneck may be re-formed at the storage device entrance, and the real end-to-end performance predictability cannot be realized. Further, in an application environment of a multi-tenant shared NVMe-oF (based on RoCEv2) storage system, the system faces the RNIC near-end resource competition caused by the micro-burst of different tenant I / O streams, and the tail delay amplification problem caused by the RNIC near-end resource competition. As shown in FIG. 1, embodiments of the present application provide a fine-grained QoS isolation method for NVMe-oF storage, which comprises the following steps: Figure 1

[0027] 101, acquiring system telemetry data and multi-tenant I / O request streams in a multi-tenant scenario;

[0028] In the embodiments of the present application, the current execution end acquires system telemetry data and multi-tenant I / O request streams in a multi-tenant scenario, and can also preprocess the system telemetry data and the multi-tenant I / O request streams to obtain processed system telemetry data and multi-tenant I / O request streams. The system telemetry data can include but is not limited to congestion events (such as the number and duration of PFC pause frames, ECN mark count) from a network switch, sending and receiving statistics (such as doorbell rate, WQE queue depth, sending byte number) from an RDMA network card, and queue and namespace statistics (such as active queue depth, completion queue depth) from a target NVMe storage controller. The multi-tenant I / O request stream contains read and write requests from different hosts accessing different namespaces.

[0029] ​It should be noted that, in order to effectively model the input traffic, the obtained multi-tenant I / O request stream can be optionally aggregated and classified within a time window (for example, 100 milliseconds), by analyzing the request size distribution and the arrival time interval, identifying the micro-burst characteristics existing therein, and generating a multi-tenant I / O traffic portrait for each tenant or virtual queue. The portrait can include two key performance indicators: burstiness and steadiness. The burstiness is used to quantify the ratio of peak rate to average rate of traffic in a short period of time, and the steadiness is used to measure the smoothness of traffic rate over time. These indicators will serve as the initial basis for subsequent adaptive parameter configuration.

[0030] 102. micro-slice the multi-tenant I / O request stream and import each obtained micro-slice into a corresponding virtual queue;

[0031] In the embodiment of the application, the current execution end decomposes the multi-tenant I / O request stream into a predetermined number of micro-slices (micro-slice), wherein the micro-slice is the basic execution unit for I / O scheduling, resource preemption and active surrender, rather than the traditional unit of entire I / O request or network packet. The refinement of this scheduling granularity is the basis for achieving nanosecond or sub-microsecond fine timing control. A raw, large NVMe I / O request (for example, 128KB) can be decomposed into hundreds or even thousands of micro-slices. By cutting large occupation time, high-priority or latency-sensitive I / O streams can preempt service in a very short time gap, thereby avoiding being blocked by large requests for a long time, effectively reducing tail latency. It should be noted that the size of each micro-slice can be fixed or variable, and exemplarily, the size can be configured in the range of 64 bytes to 256 bytes, which is not limited in the embodiment of the application.

[0032] In the embodiment, after micro-slicing the I / O request stream, each obtained micro-slice is imported into a corresponding virtual queue (Virtual Queue). The virtual queue is an abstract entity used to logically isolate different tenants or I / O streams, and can be mapped to one or more physical RDMA queue pairs (QP).

[0033] 103. determining a two-dimensional credit vector for controlling each virtual queue based on the system telemetry data;

[0034] In this embodiment of the invention, the current execution terminal determines the two-dimensional credit vector of each virtual queue based on the system telemetry data obtained in step 101. The two-dimensional credit vector is the core data structure for QoS control, containing two independent yet collaborative components (rate credit and burst credit). Specifically, rate credit... Credit This is used to constrain the average I / O release rate of a virtual queue over a longer time window (e.g., hundreds of microseconds to milliseconds) to prevent it from excessively consuming bandwidth for extended periods. Burst credits... Credit This is used to limit the maximum I / O limit that a virtual queue can continuously allow within a short or very short time window (e.g., tens of microseconds). Its main purpose is to suppress instantaneous network congestion caused by micro-bursts. For example, the generation process of the two-dimensional credit vector is dynamically calculated based on system telemetry data, especially network congestion intensity, and this embodiment of the invention does not impose specific limitations.

[0035] 104. Based on the two-dimensional credit vector of each virtual queue, schedule and gate the micro-slices in each virtual queue on a preset time grid.

[0036] In this embodiment of the invention, the current execution end schedules and gates microslices in the virtual queues based on the two-dimensional credit vectors of each virtual queue obtained in step 103, within a preset time grid. The preset time grid operates at a granularity of nanoseconds or sub-microseconds. Specifically, a two-layer gating mechanism can be deployed near the transmission path of the RNIC. Microslices must pass through a gating system controlled by the two-dimensional credit vector; only when the rate credit and burst credit of their respective virtual queues have sufficient margin is the microslice eligible for release. Eligible microslices enter a time-series gating system, which arbitrates within a very fine-grained preset time grid (e.g., 5 nanoseconds to 50 nanoseconds), allowing only one microslice to pass through each grid (or time slot). Macro-rate and burst strategies are executed precisely and micro-level at every point in time, thereby achieving fine-grained QoS isolation for multi-tenant I / O flows.

[0037] In this embodiment, the scheduling decision frequency of the system is increased to hundreds of times or even higher than that of traditional packet- or request-based scheduling, thereby providing a physical basis for achieving ultra-low tail latency and fast preemption. For example, the length of a gated time slot can be set to 10 nanoseconds, and the data payload size of a microslice can be set to 128 bytes, etc., without specific limitations in this embodiment of the invention.

[0038] Further, as a refinement and extension of the above embodiment, in order to obtain a more refined granularity of the basic execution unit while ensuring the accuracy of the receiving end reorganization and data recovery, another fine-grained QoS isolation method for NVMe-oF storage is provided, as shown in Figure 2 The step of micro-slicing the multi-tenant I / O request stream includes:

[0039] 201. Identify the semantic boundaries existing in the multi-tenant I / O request stream;

[0040] In the embodiment, the current execution end identifies the semantic boundaries existing in the multi-tenant I / O request stream by parsing the multi-tenant I / O request stream. The semantic boundaries include the boundaries of NVMe commands and data segments, storage page alignment boundaries, and namespace metadata boundaries, etc., which are not limited in the embodiment.

[0041] 202. Under the premise of not destroying the semantic integrity of the request, the identified semantic boundaries are taken as the first-priority segmentation point to segment the multi-tenant I / O request stream, obtaining micro-slices.

[0042] In the embodiment, the current execution end takes the identified semantic boundaries as the first-priority segmentation point to segment the multi-tenant I / O request stream under the premise of not destroying the semantic integrity of the request, obtaining micro-slices. That is, the multi-tenant I / O request stream is preferentially segmented at the semantic boundaries to form micro-slices. Specifically, the system checks the internal structure of an I / O request, for example, an NVMe write command usually contains a command header and a data load. At this time, the junction of the command header and the data load is a high-priority semantic boundary. Another example is that when the data load crosses the 4KB storage page boundary, the boundary point is also a preferred segmentation point. Cutting on these semantic boundaries makes it easier for the interrupted part to be logically decoupled from the sent part even if a micro-slice is interrupted due to preemption during transmission, reducing the complexity of the receiving end reorganization and data recovery, and enhancing the robustness of the system. In addition, the above cutting process effectively avoids blind cutting according to a fixed length.

[0043] 203. When the semantic boundaries are sparse, a nearest truncation strategy is adopted to determine the second-priority segmentation point to segment the multi-tenant I / O request stream, obtaining micro-slices.

[0044] In the embodiment of the present application, when the identified semantic boundaries are sparse, the current execution end can also use the nearest truncation strategy to segment the multi-tenant I / O request stream. The nearest truncation strategy means that when the semantic boundaries are sparse and the target micro-slice size cannot be met at the boundaries, the system will be truncated at the nearest position to the current segmentation point. The specific execution sequence is: first, try to segment at the semantic boundaries, if the boundaries are too sparse, then use the nearest truncation, and finally use the minimum granularity bottom-up strategy.

[0045] In the embodiment, the micro-slice is not only a segment of data load, but also a well-defined data structure, which usually includes a micro-slice header and a data payload. The micro-slice header is used to carry the metadata required for fine scheduling and reliable recombination, and the specific composition can include an original request identifier (OriginalRequest_ID): a unique identifier indicating to which upper-layer I / O request the micro-slice belongs; a slice sequence number (SliceSequence_Number): an integer starting from 0 and increasing, used to identify the order of the micro-slice in the original request; a total number of slices (TotalSlices): indicating the total number of micro-slices into which the original request is divided; a flag bit (Flags): a set of binary flags, such as START_OF_MESSAGE (indicating the first micro-slice) and END_OF_MESSAGE (indicating the last micro-slice), used to assist the receiving end in fast recombination; a data checksum (Checksum): such as CRC32, used to check the integrity of the data payload during transmission.

[0046] In a preferred implementation, the generation of a micro-slice includes: receiving an I / O request to be sent and parsing its structure to identify semantic boundaries; initializing the slice sequence number to 0 and calculating the total number of slices according to the total size of the request and the preset target size of the micro-slice (e.g. 128 bytes); starting iteration from the head of the data load of the request; at each segmentation, preferentially segmenting at the nearest semantic boundary to the current segmentation point and not exceeding the maximum size of the micro-slice, if there is no semantic boundary in the search range, segmenting according to the preset target size; creating a micro-slice header for each generated data payload, and filling the metadata of the original request identifier, slice sequence number, and total number of slices of the current context into it to form a complete micro-slice, and the slice sequence number is incremented by one after this step. Record the key metadata of the newly generated micro-slice (especially its sequence number and offset in the original request) in the slice verification index table. The table takes the original request identifier as the primary key and maintains the complete and traceable mapping relationship between the original request and all its micro-slices, which is the basis for reliable recombination and recovery after preemption.

[0047] It should be noted that, in order to ensure the consistency of data after preemption and reorganization, the system can also generate a check summary (such as CRC32) for each micro slice, and record it in a slice check index table along with its original request sequence number and in-slice sequence number for verification by the receiving end. In a preferred implementation, the construction process of the slice check index table can be: reading the virtual queue gating parameter set and the multi-tenant I / O request stream, parsing the command header, data segment and page alignment relationship of each outbound request to obtain a slice boundary candidate set; based on the slice boundary candidate set and the micro slice size configuration in the virtual queue gating parameter set, generating continuous variable-length slices for each request to obtain a micro slice sequence stream (including a virtual queue identifier and a micro slice sequence number); reading the original sequence information of the micro slice sequence stream and the multi-tenant I / O request stream, generating a check summary and a rearrangement index for each slice to obtain a slice check index table.

[0048] In another preferred implementation, the construction process of the slice check index table can be: reading the virtual queue gating parameter set and the multi-tenant I / O request stream, parsing the command header and data layout of the request, marking the start and end of the command and data segment, 512B or 4KB page alignment points and namespace metadata boundaries to obtain semantic boundary marking results; reading the semantic boundary marking results and the micro slice size configuration in the virtual queue gating parameter set, and preferentially dividing at semantic boundaries; when the boundaries are sparse, using a nearest truncation strategy to obtain a micro slice sequence stream; reading the original sequence information of the micro slice sequence stream and the multi-tenant I / O request stream, generating a check summary (length, offset summary) and a rearrangement index for each slice to obtain a slice check index table.

[0049] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to improve the nonlinear control capability of the system to cope with congestion events of different intensities, another fine-grained QoS isolation method for NVMe-oF storage is provided, as shown in Figure 3 As shown in the figure, the step of determining a two-dimensional credit vector for controlling each virtual queue based on the system telemetry data comprises:

[0050] 301, based on the system telemetry data, evaluating the congestion intensity, and based on the obtained congestion intensity evaluation value, determining a target congestion interval from a plurality of preset congestion intervals;

[0051] In the embodiment of the present application, the current execution end evaluates the congestion intensity based on the system telemetry data obtained in step 101, which can be specifically: reading the congestion event sequence of the switch and the network interface and the RDMA network card telemetry and the gated release sequence by micro-slice, windowing and timestamp aligning the time sequences of multiple sources to obtain the aligned congestion raw indicators; reading the aligned congestion raw indicators, integrating the Pause number / duration, ECN proportion, packet loss and round-trip delay fluctuation into a single intensity score according to the weight, and obtaining the evaluated congestion intensity indicators according to the threshold layering. Specifically, in a fixed time window (for example, 50-200 microseconds), the priority flow control pause frame (PFC Pause) events, ECN marking proportion, packet loss count and round-trip time (RTT) fluctuation and other indicators from the switch and the network card are integrated, and the comprehensive congestion intensity indicators (Congestion Strength_Index ) are calculated by a weighted fusion algorithm. It can be understood that the selection of the 50-200 microsecond time window is a technical compromise between response sensitivity and statistical stability. If the window is too short (for example, less than 10 microseconds), the indicators will be subject to severe noise due to a single network event, causing the strategy to frequently fluctuate; if the window is too long (for example, more than 1 millisecond), the system will be slow to respond to rapidly changing congestion conditions, missing the best opportunity for intervention. The range of 50-200 microseconds can cover several typical NVMe-oF I / O interaction rounds, which is sufficient to smooth out random noise while responding quickly enough to persistent congestion (such as the initial stage of PFC storm). When multiple telemetry indicators are integrated into a single congestion intensity indicator, an exemplary implementation is to normalize each raw indicator and then perform weighted summation. For example, Index=w1 Norm(Pause Duration )+w2 Norm(ECN Ratio )+w3 Norm(RTT Jitter ); Norm() is a normalization function that linearly or logarithmically maps the raw telemetry (such as the total Pause duration, the proportion of ECN marking to the total packet number, etc.) to the [0, 1] interval; w1, w2, w3 are weight coefficients, and their sum is 1; Pause Duration is the pause duration, ECN Ratio is the explicit congestion notification proportion, and RTT Jitter is the round-trip delay fluctuation. In practice, since the PFC Pause event usually indicates more severe congestion, its weight w1 can be assigned a relatively high value, for example, 0.6, etc., which is not specifically limited in the embodiment of the present application.

[0052] In the embodiment, the target congestion interval is determined from the obtained congestion intensity evaluation value and a plurality of preset congestion intervals. The plurality of preset congestion intervals can define three or more congestion levels according to the numerical range of the congestion intensity index value. For example, the congestion intensity index value between 0 and 30 is defined as low-level congestion, the congestion intensity index value between 30 and 70 is defined as medium-level congestion, and the congestion intensity index value above 70 is defined as high-level congestion, and the embodiment of the application is not limited in this regard.

[0053] 302, obtaining a credit adjustment strategy corresponding to the target congestion interval, and determining the rate credit and the burst credit based on the credit adjustment strategy.

[0054] In the embodiment, the current execution end is configured with different credit adjustment strategies for each preset congestion interval. For example, for the low-level congestion interval, the adjustment strategy mainly stabilizes and slightly adjusts, mainly adjusts the rate credit in a small range, and maintains a relatively loose burst credit to ensure the normal performance fluctuation of the application. For the medium-level congestion interval, the adjustment strategy synchronously tightens, and the rate credit and the burst credit are synchronously adjusted in a proportion to the congestion intensity index. For the high-level congestion interval, the adjustment strategy adopts strong suppression and asymmetric intervention: the burst credit is sharply reduced to a very small value or even cleared, and the burst source is cut off at the fastest speed; and then the rate credit is quickly adjusted. This strategy prioritizes the burst traffic with the largest instantaneous impact on the network, reflecting a deep understanding of the causes of congestion. In order to avoid frequent switching of strategies near the critical point of the interval, a hysteresis comparison mechanism can also be introduced. The current execution end obtains a credit adjustment strategy corresponding to the target congestion interval, and determines the rate credit and the burst credit based on the credit adjustment strategy. Compared with the single linear or proportional adjustment, the above segmented response credit adjustment strategy can provide stronger nonlinear control capability to cope with congestion events of different intensities.

[0055] Further, as a refinement and expansion of the specific implementation of the above embodiment, in order to not only balance the credit amount in the scheduling and gate release process, but also solve the problems of more fine timing and fairness, another fine-grained QoS isolation method for NVMe-oF storage is provided, and the scheduling and gate release in step 104 are implemented through a double-layer gate mechanism.

[0056] In the embodiment, the double-layer gate mechanism includes a token gate and a phase gate. The token gate is used to cancel the credit amount of each virtual queue according to the two-dimensional credit vector, and allows the micro-slices with credit amount meeting the preset requirements to pass; and the phase gate is used to receive the micro-slices released by the token gate, and decides the final released micro-slices according to the arbitration rules including the debt compensation state at the preset time grid.

[0057] In this embodiment of the invention, a two-layer gating mechanism achieves physical or logical decoupling of quota constraints and timing constraints. Specifically, the Token Gate focuses on the logical layer of credit verification. Based on a two-dimensional credit vector, the system generates two types of tokens in real time for each virtual queue: rate tokens and burst tokens. Rate tokens are replenished to the virtual queue's token bucket at a constant rate matching the rate credit, while burst tokens are issued all at once at the beginning of each large time window, with a total amount equal to the burst credit limit. Before a microslice can enter the scheduling candidate set, it must successfully obtain one rate token and one burst token from the token bucket of its virtual queue. If either token is insufficient, the microslice will be temporarily blocked. The existence of the Token Gate ensures that any microslice entering the next level of scheduling is compliant and meets the macro-level QoS policy. The Phase Gate focuses on the physical or logical layer of timing arbitration. It receives all qualified microslices from different virtual queues that have passed through the Token Gate and makes the final release decision on a nanosecond-level time grid. The decision-making basis of the phase gate is no longer the credit limit (because the token gate has already guaranteed this), but focuses on more refined issues of timing and fairness.

[0058] Furthermore, under this two-layer gating architecture, the arbitration rules based on the phase gate include: when there are multiple candidate microslices in the same gating time slot, the service deficit of each candidate's virtual queue is compared based on the debt compensation status; only when the service deficits of multiple candidates are the same, their remaining credit limits are compared based on the two-dimensional credit vector to decide which microslice to finally release.

[0059] In this embodiment, the arbitration logic is very clear: the first priority is the equitable debt. The phase gate will prioritize the service debt. Amount The highest-ranking virtual queue's microslices are allowed to pass through to compensate for their historical service losses. Only in the extreme case where multiple candidate queues have identical service deficits (e.g., all zero) will a second-priority rule be activated, comparing the remaining burst tokens or rate token bucket fullness of each queue to select the more abundant one. This ensures that long-term fairness is an overwhelming priority.

[0060] It should be noted that the debt compensation status in the aforementioned two-layer gating mechanism is a well-defined data structure, not merely a single service debt variable. An exemplary debt compensation status structure may include: service debt, last update timestamp, status flag, and priority boosting factor. Among these, the service debt... Amount) can be a 64-bit unsigned integer in byte unit, used to precisely measure the amount of service lost by a virtual queue due to preemption; Last Update_Timestamp ) can be a high-precision timestamp recording the time when the last change (increase or decrease) of service debt occurred, which is a key input for the debt aging process to calculate the decay amount; Status Flag ) can be an enumerated variable to describe the current state of debt, such as NO DEBT DEBT (no debt), ACCRUING (debt accumulation in progress), COMPENSATING (compensation in progress), AGING (aging in progress), to guide the scheduler to make more refined decisions; Priority Boost_Factor ) can be a floating-point number mapped by a service debt through a non-linear function (such as a logarithmic function). The scheduler can directly use this pre-calculated factor when arbitrating, rather than directly comparing the original debt value that may differ greatly, thus simplifying hardware implementation and enabling more flexible priority policies. For example, Priority Boost_Factor = 1.0 + log(1 + Debt Amount ).

[0061] Further, as a refinement and extension of the above embodiment, in order to make the system resources be effectively utilized at all times and be able to quickly respond to congestion control instructions, another fine-grained QoS isolation method for NVMe-oF storage is provided, as shown in Figure 4 The steps of scheduling and gating release of the micro-slices in each virtual queue on a preset time grid, comprising:

[0062] 401. In the scheduling process, when the micro-slice of any virtual queue is interrupted due to preemption or yielding, the service debt of the interrupted virtual queue is updated based on the size of the interrupted micro-slice; and the debt compensation state of the interrupted virtual queue is updated;

[0063] 402. In the subsequent arbitration, the scheduler raises the release priority of the interrupted virtual queue based on the debt compensation state, so that the interrupted virtual queue is compensated for the service loss.

[0064] In the embodiment of the present application, when any virtual queue is interrupted in service due to burst credit exhaustion or its channel state being in isolated channel, other virtual queues are allowed to preempt its time slot, and the corresponding preemption event is recorded to trigger the update of the debt compensation state. Specifically, the trigger condition of preemption is explicit and based on a preset strategy. For example, in a certain nanosecond-gated time slot, virtual queue VQ_A should obtain service, but the system detects that its burst credit quota has been reduced to zero within the window, or its state has been marked as being in isolated channel. At this time, the system determines that the service of VQ_A should be temporarily interrupted. Accordingly, if there is another candidate virtual queue VQ_B with a release qualification in the same time slot, the scheduler will allow VQ_B to preempt the time slot originally belonging to VQ_A, and release a micro slice of VQ_B. At the same time, the system generates a preemption record containing the preemption party (VQ_A), the preempting party (VQ_B), the occurrence time, and the interrupted micro slice information. This makes the system resources always be effectively utilized, and can quickly respond to congestion control instructions.

[0065] In addition, when the preemption event described in the previous paragraph occurs, the system reads the size of the interrupted micro slice (for example, 128 bytes) in the preemption record, and adds this value to the service arrears variable of the preemption queue VQ_A. Thereafter, in each gated time slot for release arbitration, the scheduler will take the service arrears of each candidate virtual queue as a key and usually the highest priority decision basis. For example, in the case that the credit quotas of multiple candidate queues are sufficient, the queue with the highest service arrears will win the service right of the time slot. When a preemption queue (such as VQ_A) regains service through this high-priority compensation and successfully releases a micro slice, its service arrears will be reduced by the size of the released micro slice, and will be zeroed. Fairness is transformed from a vague concept to an accurately measurable engineering implementation.

[0066] It should be noted that, in order to avoid long-term impact of historical service arrears on current scheduling priority, the service arrears recorded in the debt compensation state can also be aged, including: adopting a decay strategy to reduce the value of the service arrears recorded in the debt compensation state, to obtain a decayed service arrears; the decay strategy is one of a time-lapse decay strategy, a decay function strategy, and an event-related decay strategy; updating the debt compensation state of each virtual queue based on the decayed service arrears, so as to determine the scheduling priority of each virtual queue based on the updated debt compensation state.

[0067] In this embodiment, it is found that a large service debt accumulated a long time ago should not indefinitely affect current scheduling decisions, especially when the virtual queue has been inactive for a long time after accumulating the debt. To this end, the system can employ a time decay strategy. For example, the system can set a fixed decay period (e.g., 10 milliseconds), and at the end of each period, multiply the service debt of all virtual queues by a decay factor (Decay Factor ) less than 1, such as 0.95. Debt Amount_new = Debt Amount_old 0.95, where Debt Amount_old is the historical service debt, and Debt Amount_new is the updated service debt. In this way, the weight of the historical debt decreases exponentially over time, allowing the scheduling priority to more accurately reflect the recent fairness status. As an optional solution, the aging process can also be tied to events, for example, only when a virtual queue has not generated any I / O requests for N consecutive scheduling windows, a one-time large reduction of its service debt is performed.

[0068] In a preferred embodiment, to coordinate the potential conflict between debt compensation and credit limit, there is also a feedback mechanism between the scheduling module and the credit calculation module. When the scheduling module detects that a virtual queue is in a locked state (i.e., although it has a high compensation priority, it cannot be released due to lack of credit) with high service debt and low credit limit, it will actively send an internal signal to the credit calculation module to make a recovery suggestion. The signal will prompt the credit calculation module to appropriately increase the credit recovery speed for the specific queue in the next round of credit allocation, helping it to get out of the lock and start to clear its service debt. The specific implementation of the embodiment of the present application is not limited.

[0069] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to improve the utilization efficiency of credit resources while not destroying the long-term fairness isolation between tenants, another fine-grained QoS isolation method for NVMe-oF storage is provided, as shown in Figure 5 , which schedules and gates the release of the micro-slices in each of the virtual queues on a preset time grid, further comprising:

[0070] 501. At the end of a time window, compare the actual total amount of micro-slice release of each virtual queue with the allocated rate credit limit;

[0071] 502. If the actual total amount of micro-slice release is lower than the allocated rate credit limit, determine the corresponding virtual queue as a to-be-adjusted virtual queue with unused credit;

[0072] 503. The unused quota of the virtual queue to be adjusted is carried over to the next time window according to a preset ratio, so as to increase the sudden credit limit of the virtual queue to be adjusted in the next time window.

[0073] In this embodiment of the invention, at the end of a time window, if the total amount of actual microslices released by any virtual queue is lower than the rate credit limit allocated to that virtual queue, it is determined that there is unused credit. The unused credit is carried over to the next time window according to a preset ratio to increase the burst credit limit of the virtual queue in the next time window. Specifically, at the end of each credit assessment window (e.g., 100 microseconds), the system calculates the VQ of each virtual queue. i Actual number of bytes released (Used) Bytes _ i If the actual number of bytes used is... Bytes _ i Less than its allocated rate credit limit Crediti The difference is Unused Bytes _ i = Rate Crediti -Used Bytes _ i This refers to unused credit. The system will allocate a portion of this unused credit (e.g., Unused) Bytes _ i 0.5 (where 0.5 is the carry-over ratio) is accumulated to VQ i Burst in the next window of sudden credit Crediti Within the upper limit. To prevent unlimited accumulation leading to a potential credit bomb, the carryover amount is usually capped; for example, the accumulated carryover burst credit cannot exceed the average size of a standard I / O request. This embodiment can improve the utilization efficiency of credit resources, especially beneficial for applications with stable traffic and occasional small bursts. It is understood that in a multi-tenant environment, the traffic peaks of different tenants are usually staggered and do not occur simultaneously. Traditional credit allocation methods that strictly clear credits by window can cause a tenant to waste its allocated bandwidth quota (i.e., unused quota) during its own traffic trough, and these quotas cannot be utilized by other tenants that may be at their peak. The credit carryover mechanism in this embodiment allows a well-behaved tenant (i.e., consistently below its rate credit) to store a small portion of unused quota, converting it into a pass for future short-term bursts (i.e., increased burst credit). Essentially, it smooths the resource demand of individual users over time, making the total bandwidth demand curve of the entire system smoother, thereby accommodating more tenants or higher overall throughput with the same physical bandwidth.

[0074] It should be noted that, since the carried-over quota has an upper limit and can only be used for burst credit, it does not change the long-term average rate of the tenant (the rate credit remains unchanged), thus does not destroy the long-term fairness isolation between the tenant and other tenants, and is an optimization that finely balances between determinism guarantee and resource efficiency.

[0075] Further, as a refinement and extension of the above embodiment, in order to weaken the impact of high-risk events on shared resources without completely disconnecting services, another fine-grained QoS isolation method for NVMe-oF storage is provided, as shown in Figure 6 The method comprises:

[0076] 601, monitoring the congestion intensity reflected by the system telemetry data; when the congestion intensity triggers a high-risk threshold, determining the corresponding virtual queue as a high-risk virtual queue; and migrating the high-risk virtual queue into a preset isolation channel and updating the channel state;

[0077] In the embodiment of the application, in addition to triggering the generation of the two-dimensional credit vector in a fixed time window, it can also be triggered by identifying high-risk events. Specifically, the system telemetry data is monitored to identify a preset high-risk event indicating a sharp change in congestion condition; the preset high-risk event at least includes a priority flow control (PFC) storm or a high-density pause frame event; when the high-risk event is identified, the reevaluation and update of the two-dimensional credit vector is triggered immediately without waiting for the current time window to end. In this embodiment, the event-driven mechanism gives the system the ability to respond quickly to network emergencies. For example, when the occurrence frequency of PFC pause frames exceeds the high-risk threshold within a few microseconds, the system will immediately determine that a PFC storm has occurred and trigger credit reevaluation immediately. In the reevaluation, the system will quickly, even directly, clear the burst credit of the virtual queue associated with the storm and significantly reduce its rate credit, thereby performing strong intervention in the initial stage of congestion spread at a response speed much faster than the traditional slow start and slow recovery mechanisms based on RTT or fixed time windows.

[0078] In a further preferred embodiment of the application, when the congestion further deteriorates, the system will enable the isolation mechanism. Specifically, it also includes: monitoring the congestion intensity reflected by the system telemetry data; when it is determined that the congestion intensity related to any virtual queue triggers a high-risk threshold, migrating the virtual queue into a preset isolation channel and updating the channel state.

[0079] 602, in the isolation channel, increasing the burst credit quota of the high-risk virtual queue, and / or lengthening the minimum gate release interval of the high-risk virtual queue;

[0080] 603、In combination with the dynamic change of the congestion intensity, the burst credit or the minimum gate interval of the high-risk virtual queue is linearly or piecewise adjusted within a preset range.

[0081] In the embodiment of the application, in the isolation channel, the virtual queue is subjected to a more stringent burst credit or a longer minimum gate interval than the main channel, that is, an elastic mechanism is provided for the management of the isolation channel. Specifically, as the congestion intensity changes, the burst credit or the minimum gate interval is linearly or piecewise adjusted within a preset range to achieve elastic isolation.

[0082] The isolation channel is a logical state, and a virtual queue entering this state will be subjected to a set of independent and more stringent scheduling strategies. Bad neighbors that cause serious congestion can be isolated to protect the quality of service of other good neighbors in the network. For example, once a virtual queue is migrated into the isolation channel, the upper limit of its burst credit may be locked to a minimum value (such as allowing only one micro slice to be released), and the minimum time interval for releasing its micro slice is forced to be lengthened (for example, from 10 nanoseconds to 50 nanoseconds). The impact of the virtual queue on shared resources can be weakened without completely disconnecting the service.

[0083] In a further preferred embodiment of the application, the execution process of the isolation mechanism can be that when a virtual queue enters the isolation channel, its congestion intensity index is 90, at which time the system configures the most stringent parameters for it (for example, the upper limit of the burst credit is 1 micro slice, and the minimum release interval is 100 nanoseconds). If its congestion intensity later decreases to 75 (still in the high congestion interval, but has eased), the system will relax its parameters accordingly according to a preset linear function or piecewise mapping table (for example, the upper limit of the burst credit is relaxed to 2 micro slices, and the minimum release interval is shortened to 80 nanoseconds). The intensity of isolation is accurately matched with the intensity of risk, reducing system oscillation caused by parameter mutation, and enabling the virtual queue to exit or enter the isolation state more smoothly.

[0084] In a further preferred embodiment of the application, after the congestion condition is alleviated, the system will execute a recovery logic. Specifically, the two-dimensional credit vector is generated, and further comprising: after the congestion condition is alleviated, the credit recovery speed is differentially set according to the debt compensation state; specifically, a faster credit recovery slope is configured for a virtual queue with a higher service arrears to accelerate its debt repayment.

[0085] In this embodiment, the deep coupling of the fairness mechanism and the congestion control mechanism is embodied. When the congestion intensity index decreases, the system starts to restore the credit quota for the affected queue, but the speed of restoration is not the same. The system queries the debt compensation state of each queue. For those queues that have accumulated a higher service arrears due to being occupied a lot during congestion, the system allocates a steeper credit recovery curve to them, so that the rate credit and burst credit can return to the normal level faster. Those queues that make sacrifices to alleviate congestion can be compensated in the first time, and the fairness closed loop of the whole congestion control process is realized.

[0086] Further, as a refinement and extension of the specific implementation of the above embodiment, another fine-grained QoS isolation method for NVMe-oF storage is provided, which further comprises:

[0087] Fusing the two-dimensional credit vector, channel state, and NVMe controller queue and namespace statistics obtained from system telemetry data to obtain a fusion result;

[0088] In this embodiment, cross-domain information is fused to form a globally consistent scheduling view. Specifically, the fusion process collects three types of key information: first, the two-dimensional credit vector and the channel state representing network passage permission; second, the NVMe controller queue and namespace statistics reflecting the current load and service capability of the backend storage device, such as the average IOPS, average delay, and in-flight instruction number of a specific namespace. Fusion enables the network side to predict and adapt to the processing capacity of the storage device side, avoiding the situation where I / O requests still produce queuing and delay peaks at the device end even if the network is smooth.

[0089] Mapping the fusion result to the submission interval and the upper limit of the concurrency depth for each namespace to generate an aligned I / O scheduling plan that matches the target device service beat;

[0090] In the embodiment of the application, the fusion result is mapped to the submission interval and the upper limit of the concurrency depth for each namespace; specifically, the mapping process can follow a set of preset, dynamically adjustable rules. For example, the mapping of the submission interval (Submission Interval ) mainly depends on the rate credit (Rate Credit ) of the virtual queue. A reciprocal relationship model can be established: Submission Interval = K interval / Rate Credit . Wherein, K intervalis a baseline coefficient related to target device type and I / O size. When a virtual queue has a higher rate credit, it means it is allowed to submit I / O requests to the target namespace at a shorter time interval (i.e. higher frequency). For the mapping of concurrent depth upper limit (Max ConcurrencyDepth ), the decision is more complex, usually the result of the joint action of burst credit (Burst Credit ) and device real-time status. On one hand, burst credit determines the number of I / O requests that can burst in a very short window, directly limiting the peak of short-term concurrency. On the other hand, the historical average concurrency depth and completion delay of the namespace obtained from telemetry data are used to set a long-term, steady-state concurrency depth baseline to prevent excessive pressure on the device queue.

[0091] In a specific implementation, the system maintains a sliding window for each namespace to count the number of in-flight I / Os. When planning to submit a new I / O request, the system checks whether the number of in-flight I / Os will simultaneously break through the instantaneous upper limit determined by the burst credit and the steady-state upper limit determined by the device status after submitting the request.

[0092] In the embodiments of the present application, an aligned I / O scheduling plan that matches the target device service tempo is generated. Specifically, based on the above mapping calculation, the system generates a dynamic and quantitative I / O scheduling plan for each namespace. This plan is not just a simple rate limit, it contains multi-dimensional instructions such as submission tempo, batch size, concurrency limit, etc. Its overall rhythm matches the current service capacity and tempo of the target NVMe device, achieving the same frequency resonance.

[0093] According to the aligned I / O scheduling plan, a gating and limiting instruction is generated, and the gating and limiting instruction is written back to the scheduling and gating release mechanism.

[0094] In the embodiments of the present application, a feedback loop is constructed, so that the macroscopic scheduling plan can be accurately executed by the microscopic gating mechanism. Specifically, the gating and limiting instruction can directly act on the gating arbitration logic. For example, the instruction about the concurrency depth upper limit N in the scheduling plan can be translated into a specific gating rule: for the virtual queue belonging to the target namespace, if the total number of I / O requests corresponding to the micro-slices that have been released and have not received completion confirmation reaches N at any time, any new micro-slice belonging to it is temporarily prohibited from entering the time slot candidate set. The upper-level, I / O request-oriented scheduling plan is seamlessly and compulsorily implemented to the lower-level, micro-slice-oriented gating execution level, ensuring the consistency and effectiveness of the strategy.

[0095] As an optional enhancement, the scheduling and gating release mechanism can also contain an exception fallback strategy. If the system monitors that the end-to-end latency of a certain namespace suddenly abnormally increases during the execution of the aligned scheduling plan, even if its credit and concurrency are within the plan, the fallback strategy will be triggered. After triggering, the system will temporarily impose a more conservative scheduling strategy on the namespace than the plan (for example, actively lengthen the submission interval, reduce the concurrency upper limit) to cope with unknown backend storage jitter, which embodies the robustness of the scheme.

[0096] It can be understood that the device service beat refers to the characteristic mode of the time required by the target NVMe storage device (specific to a certain namespace) to process and complete a unit I / O request (for example, a 4KB random read request). It is not only an average value, but also contains its distribution characteristics. A specific quantification method is that the system establishes an I / O completion time histogram for each namespace by continuously monitoring the completion queue telemetry data from the NVMe controller. The mode or P50 (median) delay of the histogram can be regarded as the core service beat (Service Rhythm_Period ) of the namespace. For example, it is calculated that the core service beat of a certain namespace is 30 microseconds. The alignment refers to that the submission interval (Submission Interval ) of the generated I / O scheduling plan should be adjusted to an integer multiple or a harmonious fraction of Service Rhythm_Period , and the instantaneous concurrency depth should be limited to the order of Submission Interval / Service Rhythm_Period . The synchronization of the I / O production rhythm on the host side and the I / O consumption rhythm on the device side avoids the generation of additional queuing delay at the device queue entrance due to the beat misalignment, thereby controlling the end-to-end delay at the level closest to the physical limit of the device.

[0097] Further, as a refinement and extension of the specific implementation manner of the above embodiment, in order to maintain the phase stability between adjacent virtual queues without cross-device clock synchronization, another fine-grained QoS isolation method for NVMe-oF storage is provided, which comprises:

[0098] Maintain an independent virtual time axis for each virtual queue and record it in the virtual time phase state; when the micro-slice of any virtual queue is released, advance its corresponding virtual time axis according to the preset gating beat; based on the RDMA network card telemetry contained in the system telemetry data, correct the phase of the virtual time axis to maintain the phase stability between adjacent virtual queues without cross-device clock synchronization.

[0099] In this embodiment, a virtual time phase locking mechanism is implemented. Specifically, the system implements a virtual time phase locking mechanism for each virtual queue (VQ). i Maintain a virtual time variable VT i Whenever VQ i After a microslice is successfully released in a certain physical time slot, its virtual time VT i This advances by a fixed tick amount (e.g., equal to the length of the gated time slot). Ideally, the virtual times of all active virtual queues should be roughly in sync. However, due to slight variations in hardware processing, phase drift may occur. In this case, the system compares telemetry data from the local RDMA network card (such as the actual depth of the WQE transmission queue) with the expected model. If VQ is detected... i If the actual transmission progress of a virtual queue consistently exceeds or lags behind the advancement of its virtual time, the system applies a small phase correction value (positive or negative) to the VTi, pulling it back to the average virtual time of all queues. The virtual time phase-locked mechanism uses local high-frequency telemetry as a reference, achieving coordination and stability of multiple logical queues at the micro-timing level without requiring any external high-precision clock synchronization signals, effectively avoiding performance degradation caused by accumulated timing jitter. For example, the real-time depth of the hardware transmit queue (WQE queue) of an RDMA network card can be considered an observed variable of the system state. In an ideal system perfectly synchronized with virtual time, the queue depth should be stable at a small level fluctuating around zero. When the actual transmission behavior of a virtual queue precedes its virtual time (i.e., too fast release), requests accumulate in the hardware queue, increasing its depth; conversely, if its behavior lags (too slow release), the queue depth tends to remain consistently zero. Therefore, the deviation of the queue depth from its ideal value (e.g., a depth of 1) is significant. Depth - Target Depth This constitutes the error signal for negative feedback control, where Actual Depth Target represents the actual queue depth. Depth This represents the ideal queue depth. The minute phase correction is based on the control output of this error signal. A preferred implementation is to use a proportional-integral (PI) controller: Correction = Kp Error + Ki The function `Integral(Error, dt)` calculates the correction value. `Kp` is the scaling factor, `Ki` is the integration factor, and `dt` is the time step in the integration calculation. The calculated correction value `Correction` is added to or subtracted from the virtual time variable `VTi` of the corresponding virtual queue in tiny steps (e.g., nanoseconds), thereby continuously and smoothly pulling its phase back onto the correct track in coordination with other queues, achieving self-stabilized phase locking without the need for an external clock.

[0100] The embodiment of the present application provides a fine-grained QoS isolation method for NVMe-oF storage. Compared with the prior art, the present application obtains a multi-tenant I / O request stream, and performs micro-slicing processing on the multi-tenant I / O request stream, so that a basic execution unit with more refined granularity is obtained. By obtaining system telemetry data in a multi-tenant scenario, and determining a two-dimensional credit vector for controlling each virtual queue based on the system telemetry data, and based on the two-dimensional credit vector of each virtual queue, the micro-slices in each virtual queue are scheduled and gated on a preset time grid, which can effectively alleviate the RNIC near-end resource competition of the multi-tenant shared NVMe-oF storage system caused by the micro-burst of different tenant I / O streams, and further effectively avoid the problem of tail delay amplification, thereby improving the service quality and predictability in a shared storage environment.

[0101] Further, in order to cope with extreme burst traffic, the following steps are further included before scheduling and gating: monitoring the doorbell rate of each virtual queue in the system telemetry data, and when the doorbell rate of any virtual queue increases by more than a preset threshold, determining that it is a high-burst queue; temporarily allocating additional hardware queue pairs to the high-burst queue from a shared hardware queue pair pool, and recording the allocation strategy in the gating parameter set of the virtual queue to guide subsequent hardware resource mapping.

[0102] In the embodiment, a dynamic hardware resource hot migration strategy is implemented. The doorbell rate is a direct indicator of the activity of a queue submitting tasks. When the system monitors that the doorbell rate of a certain virtual queue increases by more than 50% in a short time (for example, within 100 microseconds), it can be determined that it is experiencing a severe traffic burst. In order to avoid its own hardware queue becoming a bottleneck, the system will temporarily add one or more QPs from a reserved shared queue pair (Queue Pair, QP) pool to it, and update its mapping relationship. On the one hand, it provides a wider physical channel for the burst queue, and on the other hand, it avoids the cross-queue negative impact of the burst queue on other queues on the RNIC due to its own QP congestion.

[0103] According to one aspect of the present application, the gating parameters can also be adaptively determined through cluster analysis of burstiness and activity. Specifically, high burstiness alone may only be an occasional and short-lived event, while high activity (for example, measured by doorbell rate or total I / O volume) alone may only be a smooth large flow. Multiplying the two, the composite index Composite Index = Burstiness Activity, which can more accurately identify those virtual queues that continuously generate high-intensity bursts and pose the most severe threat to the network. Both the intensity and frequency of the bursts are taken into account. Based on this combined index, the system can automatically divide all virtual queues into different clusters, such as high-risk burst cluster, smooth heavy traffic cluster, and low-load cluster, using a simple clustering algorithm such as K-means. The system can then apply differentiated and optimal gating parameters to different clusters: configure smaller micro-slices and shorter gating slots for the high-risk burst cluster to achieve the strongest preemption granularity and response speed; configure larger micro-slices and longer slots for the smooth heavy traffic cluster to improve transmission efficiency and reduce CPU overhead. This enables the QoS policy to evolve from a one-size-fits-all or simple one-to-one configuration to more intelligent group-based fine-grained management.

[0104] According to another aspect of the present application, the calculation process of the two-dimensional credit vector can also be: the accurate calculation of the rate credit can be implemented using a variant of a token bucket algorithm. The system maintains a rate credit bucket Rate Credit_Bucket for each virtual queue, with an upper limit on the capacity of Rate Credit_Max . The bucket is filled with credit at a constant replenishment rate Replenish Rate . The value of Replenish Rate corresponds to the average release rate linked to the congestion intensity. After a time interval delta t , the available amount in the rate credit bucket is updated as follows: Rate Credit_Available = min(Rate Credit_Max , Rate Credit_Previous + Replenish Rate delta t ), where Rate Credit_Previous is the rate credit value at the previous time. The size of each micro-slice is deducted from this bucket when it is released. The initial value of the burst credit is determined based on the burstiness index in the multi-tenant I / O traffic profile. Specifically, the initial burst credit amount Initial Burst_Credit can be a function positively related to the historical burstiness of the queue Burstiness Metric and is constrained by the RNIC hardware cache size System Buffer_Size . For example: Initial Burst_Credit = min(System Buffer_Size 0.1, Base Burst_Credit + k burst Burstiness Metric ); where BaseBurst_Credit is the system-wide reference burst allowance, k burst is the gain coefficient. The queues that historically exhibit stronger burstiness will be granted higher, but still within the safety range, initial burst allowance.

[0105] The interaction between the two dimensions of rate credit and burst credit, in addition to the unused rate credit being transferred to burst credit, is also reflected in the linkage suppression mechanism under global congestion. For example, when the system detects a global and extremely high-risk congestion event such as a PFC storm, the top-level adaptive module can issue an emergency braking instruction. This instruction will simultaneously act on both credit dimensions, for example, not only will the burst credit be cleared, but the replenishment process of the rate credit will also be temporarily suspended (i.e., Replenish Rate is temporarily 0) until the congestion event is initially alleviated. This allows the system to take the most decisive coordinated suppression measures in the most critical situation.

[0106] In summary, the present application discloses a fine-grained QoS isolation method for NVMe-oF storage, which includes: acquiring system telemetry data and multi-tenant I / O request flow; decomposing the multi-tenant I / O request flow into micro-slices that can be scheduled, preempted, and yielded on a nanosecond or sub-microsecond time grid; generating a two-dimensional credit vector containing rate credit and burst credit based on system telemetry data to separate and control average rate and instantaneous burst; and gating the micro-slices according to the two-dimensional credit vector. A service shortage record and debt compensation mechanism is also introduced to ensure long-term fairness, and a micro-pause isolation channel is used to suppress high-risk congestion such as PFC storm. Further, by mapping the credit allowance on the network side to the submission interval and concurrent depth on the target device side, the end-to-end service beat is aligned. This can effectively suppress micro-bursts in the RNIC near-end, cut off the delay amplification path, reduce I / O tail latency, and improve the service quality predictability in a shared storage environment.

[0107] In a specific embodiment, there are two virtual queues VQ1 (stable Web service) and VQ2 (bursty big data analysis) in the system. The calculation of the congestion intensity index is simplified as: Index = 10 Pause Events_Per_100us(That is, a Pause event occurs every 100 microseconds, and the index increases by 10). The segmented response thresholds are: Low (Index < 30), Medium (30 ≤ Index < 70), High (Index ≥ 70). At the initial time T0, the network condition is good, and the congestion intensity index (Index) on both queue paths is 10 (low congestion). Based on the initial I / O profile, the assigned two-dimensional credits (rate credit, burst credit) are: VQ1 (100KB, 20KB) and VQ2 (80KB, 50KB). The service debt for both parties... Amount All values ​​are 0. At time T1, the VQ2 application starts a large-scale computational task, generating dense I / O bursts, causing its network path to experience 9 consecutive Pause events within 100 microseconds. At this time, the congestion intensity index of the VQ2 path is 0. VQ2 Soaring to 10 9 = 90. Because of Index VQ2 =90≥70, triggering a high congestion event. Under high congestion, VQ2 traffic was strongly suppressed, resulting in a preemption of a total of 512 bytes of microslices in an I / O request that was being transmitted. VQ2 service debt. AmountVQ2 The value changes from 0 to 512 bytes. The system immediately implements the high-congestion interval credit adjustment strategy for VQ2: VQ2's burst credit adjustment. CreditVQ2 The rate was immediately reduced from 50KB to a minimum, such as 2KB, to immediately terminate its burst behavior; VQ2's rate credit. CreditVQ2 It was drastically reduced; for example, by applying a penalty factor of 0.2, it became 80KB. 0.2 = 16KB; due to Index VQ2 When the index is ≥70, the system marks VQ2 as isolated and applies a longer minimum gating interval. At time T2, due to the strong intervention of the credit mechanism, the traffic of VQ2 is effectively controlled, the congestion on its path is alleviated, and the number of pause events within 100 microseconds is reduced to 5. At this time, the index... VQ2 Decreased to 10 5 = 50. Since 30 ≤ Index VQ2 <70, congestion level drops from high to medium. The system implements a new strategy for VQ2: VQ2 remains isolated, but its restrictions are relaxed according to the resilience mechanism. For example, its minimum gating interval, based on Index=50, is linearly adjusted back to 80 nanoseconds from a maximum of 100 nanoseconds; the system begins to restore credit for VQ2 because a service deficit (Debt) has been detected. AmountVQ2= 512 > 0, the system will apply an accelerated credit recovery slope to it. Assume the base credit recovery rate is +10KB per window, and the debt acceleration factor is 0.02 Debt Amount . Then the amount of credit recovered by VQ2 in this window is 10KB + (0.02 512)B = 10KB + 10KB = 20KB. Its rate credit is recovered from 16KB to 16KB + 20KB = 36KB. A debt-free queue, in this case, can only recover to 16KB + 10KB = 26KB. At time T3, the congestion subsides, Index VQ2 drops to 15 (low congestion). The system moves VQ2 out of the isolation channel and restores its normal state. Its credit continues to recover, and since it still holds a service deficit, it will get higher priority in the gated arbitration with VQ1 and have a chance to release more micro-slices and gradually clear its 512-byte debt.

[0108] In one possible implementation, the entire QoS isolation method is placed in a continuous evaluation and feedback loop. This loop shifts the focus from intermediate process metrics (such as network congestion, credit limit) to the final, user-perceptible quality of service metric, i.e., the end-to-end completion delay of I / O requests, especially the high-percentile tail delay (such as P99, P99.9). Specifically, this adaptive loop includes the following steps:

[0109] While executing the aligned I / O scheduling plan, the system continuously monitors the end-to-end I / O performance of each virtual queue (or its corresponding namespace). In a specific implementation, the system can collect the complete life cycle duration of each I / O request from submission to the final completion acknowledgment (CQE, Completion Queue Entry) from the storage driver layer or the host-side initiator. By performing statistics on these duration data within a sliding time window, the system can calculate the P99, P99.9, and other key tail delay indicators of each virtual queue. Compare these real-time observed tail delay values with the target tail delay threshold set for this virtual queue in the initial modeling phase.

[0110] When the above comparison result appears a preset abnormal pattern, a risk control mechanism will be triggered. The triggering condition can include but is not limited to: chronic degradation: the P99.9 tail latency of a certain virtual queue continuously exceeds a certain proportion (for example, more than 20%) of its target threshold in a plurality of consecutive evaluation periods; acute spike: the tail latency of a certain virtual queue jumps by more than a preset multiple (for example, 5 times the recent moving average) in a single evaluation period. Once the risk control event is triggered, the global adaptive module immediately generates and issues high-priority rapid intervention instructions to override the current regular QoS strategy driven by the underlying signals. These instructions can include: temporary credit tightening instructions, which are sent directly to the credit calculation module, which will force the module to apply a temporary credit policy that is more stringent than the one calculated based on the current congestion intensity to the virtual queue that triggered the abnormality, for example, temporarily clearing its burst credit and halving its rate credit. This is a top-level veto based on the final effect, used to deal with the situation where the underlying congestion signal fails to accurately reflect the real performance degradation of the application. Temporary preemption priority adjustment instructions, which are sent directly to the scheduling and gating module, which can temporarily adjust the arbitration logic, for example, forcibly lower the base priority of the abnormal virtual queue in the phase gate, making it more likely to be preempted by other normal queues; or temporarily suspend compensation for its service arrears to avoid it continuing to occupy too many resources in the case of performance abnormalities.

[0111] Rapid intervention is a short-term hemostatic measure. In order to realize the long-term self-optimization of the system, the system will record the context of each risk control event in detail, including the traffic profile at that time, the triggered delay data, the adopted intervention measures and the performance change after the intervention, forming a tail latency evaluation report. The report will be used to drive a long-period baseline re-estimation process. Specifically, the system will analyze the patterns in the report and generate adaptive tuning instructions to feed back to the underlying telemetry modeling module. For example, if the analysis finds that a certain virtual queue frequently causes tail latency spikes due to burst traffic, even if its average rate is not high, the adaptive tuning instruction may suggest that the underlying telemetry modeling module increase the burst degree index baseline of the queue during the next modeling, so that the credit calculation module will naturally allocate a more restrictive burst credit policy to it that is more in line with its actual behavior pattern. For another example, if a queue's tail latency continues to fail to meet the target threshold even in the absence of congestion and sufficient credit, it may mean that the initially set target tail latency threshold is not realistic. At this time, the adaptive tuning instruction may suggest adjusting its target threshold to a more reasonable range, and may be accompanied by a system alarm to prompt the operation and maintenance personnel to check whether there is a bottleneck in the backend storage device.

[0112] This embodiment elevates the QoS isolation mechanism from a passive, network-physical-signal-based execution system to an intelligent system with proactive risk control, self-learning, and long-term optimization capabilities by introducing a top-level monitoring and feedback loop based on end-user experience (tail latency). This ensures that the application is not only theoretically sound but also effectively and continuously guarantees service quality in multi-tenant environments.

[0113] According to one aspect of this application, acquiring system telemetry data and multi-tenant I / O request flows may further include a more specific process of micro-burst feature extraction and virtual queue descriptor initialization. Specifically, after the system performs high-frequency sampling (e.g., every microsecond) of RDMA network card telemetry (especially doorbell rate and WQE queue depth), it employs a sliding window-based peak detection algorithm. When the instantaneous value of the doorbell rate within the window exceeds a specific multiple (e.g., 3 times) of its long-term moving average, and simultaneously the WQE queue depth shows a rapid increase, the system marks this event as a micro-burst feature and records its peak value, duration, and other parameters. These identified micro-burst features are used to calculate the burst index for each virtual queue. Furthermore, the initial virtual queue descriptor list is not merely a simple list of queues; each item is a data structure containing rich metadata. In addition to tenant and namespace identifiers, the structure also includes a priority baseline, which can be pre-set according to the service SLA (Service Level Agreement) and serves as the basis for all subsequent scheduling arbitrations. At the same time, it also includes a target tail latency threshold (e.g., P99.9 < 500 microseconds), which is the final criterion for the global adaptive tuning loop to perform performance evaluation and risk control triggering.

[0114] In one embodiment of this application, the physical resource mapping mechanism for virtual queues to RDMA queue pairs can also be as follows: the system maintains a global VQ-to-QP mapping table, which uses the virtual queue identifier (VQ_ID) as the primary key and records the identifiers of one or more physical QPs currently associated with each virtual queue. During the initialization phase, the system can employ a hash-based static mapping algorithm. For example, the VQ_ID is hashed, and the result is moduloed by the number of QPs in a default, system-reserved public QP pool, thereby allocating an initial physical QP to each VQ. QP =Public QP_Pool [hash(VQ_ID) % num public_qps ], where Assigned QP For physical queue pairs assigned to virtual queues, Public QP_PoolThe public queue pair reserved for the system to the resource pool, hash(VQ_ID) represents performing a hash function on the virtual queue identifier, represents a modulo operation, num public_qps The number of queue pairs in the pool for the public queue pair. The dynamic adjustment strategy of the mapping relationship, that is, QP pool live migration, has the following specific process: the system continuously monitors the doorbell rate of each VQ. When the rate of a VQ increases by more than a threshold (for example, 100%) in T time (for example, 100 microseconds) and the WQE queue depth of the QP currently associated with it is continuously at a high level, the dynamic adjustment is triggered. From a dedicated, shared hardware QP pool, one or more additional QPs are allocated to the VQ according to the current available resources. The newly allocated QP identifier is appended to the entry corresponding to the VQ_ID in the VQ-to-QP mapping table. Thereafter, all new micro-slices sent to the VQ will be distributed on the multiple QPs associated with it. The distribution strategy can be a simple round robin or a hash based on some header fields (such as sequence number) of the micro-slice to ensure that the micro-slices of the same I / O request fall on the same QP as much as possible, reducing out-of-order. When the load of the VQ is monitored to fall back to the normal level and continues for a period of time, the system will perform the reverse operation to gracefully drain the temporarily allocated QP connection, and then release it back to the shared pool and update the mapping table.

[0115] In another embodiment of the present application, a fine-grained QoS isolation method for NVMe-oF storage, the method further comprises: based on the multi-tenant I / O request stream, the congestion event sequence, the RDMA network card telemetry, and the NVMe controller queue and namespace statistics, establishing a multi-tenant I / O traffic portrait and an initial virtual queue descriptor list; according to the multi-tenant I / O traffic portrait and the initial virtual queue descriptor list, decomposing the multi-tenant I / O request stream into multiple micro-slices, and gating the micro-slices to generate a gated release sequence and a debt compensation state according to the micro-slices; in response to the congestion event sequence and the RDMA network card telemetry, and in combination with the gated release sequence and the debt compensation state according to the micro-slices, generating a two-dimensional credit vector and a channel state for each virtual queue, and using the two-dimensional credit vector and the channel state to guide the gated release; merging the two-dimensional credit vector, the channel state, and the NVMe controller queue and namespace statistics to generate an aligned I / O scheduling plan, and adjusting the I / O sending behavior and the gated release on the host side according to the aligned I / O scheduling plan.

[0116] In further embodiments, the generating the gated release sequence comprises: reading the micro-slice sequence stream and the virtual queue gating parameter set, selecting candidate slices with release eligibility in the current time slot according to the gating time slot configuration, to obtain a time slot candidate set; reading the time slot candidate set, the two-dimensional credit vector, the debt compensation state and the channel state, and selecting a release object in the current time slot according to the debt priority and the credit limit, to obtain a time slot release decision; and according to the time slot release decision and the slice verification index table, writing a release record on a time grid and updating a ledger, to obtain the gated release sequence according to the micro-slice. Optionally, the debt priority instruction can also be generated: reading the preemption record and the transfer record and the gated release sequence according to the micro-slice, converting the interrupted slice amount into a service arrear and entering the account, to obtain a debt entry table; reading the debt entry table and the gated release sequence according to the micro-slice, and deducting the debt according to the actual release amount in the subsequent time slot where continuous release is obtained, to obtain a debt compensation state; and reading the debt compensation state and the two-dimensional credit vector, temporarily increasing the arbitration weight of the virtual queue when the debt is high and the credit is insufficient, and sending a recovery suggestion to the credit module, to obtain the debt priority instruction. The debt priority instruction can be used for real-time sorting, and can also be used for credit recovery strategy to avoid isolation.

[0117] In another possible embodiment, the generating the gated release sequence can also comprise: reading the two-dimensional credit vector and the credit consumption ledger of the previous time window (if any), calculating the rate token and burst token available in the window for each virtual queue, to obtain a token quota table; reading the token quota table and the micro-slice sequence stream, and only allowing slices with sufficient tokens (burst token > 0 and rate token rhythm met) to enter the candidate set, to obtain a token-qualified candidate set; reading the token-qualified candidate set, the virtual queue gating parameter set, the debt compensation state and the channel state, and arbitrating in the current time slot according to the gating time slot configuration, giving priority to queues with high debt and reducing the weight of isolated channels, to obtain a time slot release decision; and reading the time slot release decision and the slice verification index table, writing a release record on a time grid and canceling the corresponding token, to obtain the gated release sequence according to the micro-slice and the token deduction record.

[0118] In an optional implementation, the congestion intensity indicator is obtained, and the congestion event sequence of the switch and the network interface and the congestion intensity indicator are read, and when a Pause storm or abnormal jump is detected, a re-evaluation is triggered once outside the window, to obtain an event re-evaluation result. Further, the generating the two-dimensional credit vector can also comprise: reading the congestion intensity indicator and the debt compensation state, and setting the average release amount of each virtual queue in a hierarchical or segmented response manner, to obtain a rate credit allocation result; reading the congestion intensity indicator and the debt compensation state and the recent release statistics of the gated release sequence according to the micro-slice, and setting the burst amount available in the short window for each virtual queue, to obtain a burst credit allocation result; and combining the rate credit allocation result and the burst credit allocation result into the two-dimensional credit vector, and delivering it to the gating and accounting modules.

[0119] Optionally, generating a two-dimensional credit vector can also involve: reading the congestion intensity index, dividing the intensity into low, medium, and high intervals according to a threshold, and recording the current interval and hysteresis state to obtain the congestion stratification result; reading the congestion stratification result and debt compensation status, fine-tuning the rate credit in the low interval, proportionally reducing it in the medium interval, rapidly reducing it in the high interval, and setting a minimum guarantee amount to obtain the rate credit allocation result; reading the congestion stratification result, the gating release sequence near-window statistics by micro-slice, and the debt compensation status, first clearing or strongly tightening the burst credit in the high interval, synchronously tightening it in the medium interval, and slightly recovering it in the low interval to obtain the burst credit allocation result; reading the debt compensation status and congestion stratification result, setting a faster rate and burst recovery slope for the high debt virtual queue in the low / medium interval, and maintaining strict control in the high interval to obtain the recovery slope parameter; combining the rate credit allocation result and the burst credit allocation result with the recovery slope parameter to synthesize (rate credit, burst credit) for each virtual queue, generating a two-dimensional credit vector and sending it to the gating and accounting module.

[0120] In an optional embodiment, the aligned I / O scheduling plan is generated, including: reading the two-dimensional credit vector, the channel state, the NVMe controller queue and the namespace statistics, mapping the available rate credit and burst credit of each namespace to the submission interval and single request batch to obtain a submission interval suggestion; reading the submission interval suggestion, the NVMe controller queue, the namespace statistics and the beat information of the micro-slice gated release sequence to generate the maximum concurrency and threshold of each namespace, and writing back the gated limiting instruction to obtain an intermediate state of the alignment strategy; reading the intermediate state of the alignment strategy and the beat information of the micro-slice gated release sequence, limiting the maximum number of in-flight requests in a short time window according to the namespace, and establishing a sliding window counter for each namespace to obtain a namespace concurrency shaping table; reading the namespace concurrency shaping table, the submission interval suggestion and the channel state, grouping and aligning the release beats of the requests of the same namespace in a short window according to the submission interval, and applying a longer inter-group minimum interval to the isolated channel namespace to obtain a beat grouping plan; reading the priority threshold and the rollback rule of the priority threshold and the rollback rule, the beat grouping plan and the namespace concurrency shaping table, generating an executable aligned I / O scheduling plan (including the submission interval, the group size, the inter-group interval, the maximum in-flight number, the priority threshold and the rollback rule of each namespace), and packaging it as a delivery interface for the host sending end and the gate controller; at the same time, output the plan delivery instruction (internal) for driving execution. Optionally, during the execution of the aligned I / O scheduling plan, the end-to-end tail delay and fairness indicators are evaluated; when the prediction confidence decreases or an abnormal peak occurs, temporary weight reduction or rollback is triggered, and a tail delay evaluation report and an adaptive parameter adjustment instruction are output.

[0121] As an implementation of the method shown in the above Figure 1 As an implementation of the method shown in the above

[0122] The acquisition module is configured to acquire system telemetry data and a multi-tenant I / O request stream in a multi-tenant scenario.

[0123] The micro-slice processing module is configured to perform micro-slice processing on the multi-tenant I / O request stream, and import each micro-slice obtained into a corresponding virtual queue.

[0124] The credit evaluation module is configured to determine a two-dimensional credit vector for controlling each virtual queue based on the system telemetry data.

[0125] The scheduling module is configured to schedule and gate release the micro-slices in each virtual queue on a preset time grid based on the two-dimensional credit vector of each virtual queue.

[0126] The embodiment of the present application provides a fine-grained QoS isolation device for NVMe-oF storage, compared with the prior art, the present application obtains a multi-tenant I / O request stream, and performs micro-slice processing on the multi-tenant I / O request stream, so that a basic execution unit with finer granularity is obtained. By obtaining system telemetry data in a multi-tenant scenario, and determining a two-dimensional credit vector for controlling each virtual queue based on the system telemetry data, scheduling and gate releasing the micro-slices in each virtual queue on a preset time grid based on the two-dimensional credit vector of each virtual queue, the RNIC near-end resource competition of the multi-tenant shared NVMe-oF storage system caused by the micro-burst of different tenant I / O streams can be effectively alleviated, and the problem of tail delay amplification is effectively avoided, and the service quality and predictability in the shared storage environment are improved.

[0127] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be realized by general computing devices, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by the computing devices, so that they can be stored in the storage devices and executed by the computing devices, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps thereof can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any particular combination of hardware and software.

[0128] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A fine-grained QoS isolation method for NVMe-oF storage, characterized in that, include: Acquire system telemetry data and multi-tenant I / O request streams in multi-tenant scenarios; The multi-tenant I / O request stream is processed into micro-slices, and the resulting micro-slices are imported into the corresponding virtual queues; Based on the system telemetry data, a two-dimensional credit vector is determined for controlling each of the virtual queues; the two-dimensional credit vector includes rate credit and burst credit. Based on the two-dimensional credit vector of each virtual queue, the micro-slices in each virtual queue are scheduled and gated on a preset time grid; The micro-slicing process for the multi-tenant I / O request stream includes: Identify the semantic boundaries existing in the multi-tenant I / O request stream, including the boundaries between NVMe commands and data segments, storage page alignment boundaries, and namespace metadata boundaries; Without compromising the semantic integrity of the request, the identified semantic boundaries are used as the first priority segmentation points to segment the multi-tenant I / O request stream, resulting in micro-slices. When the semantic boundaries are sparse, a nearest truncation strategy is used to determine the second priority segmentation point to segment the multi-tenant I / O request stream, resulting in micro-slices.

2. The method according to claim 1, characterized in that, The step of determining the two-dimensional credit vector for controlling each of the virtual queues based on the system telemetry data includes: The system assesses congestion intensity based on telemetry data and determines a target congestion interval from a set of predefined congestion intervals based on the obtained congestion intensity assessment value; the set of predefined congestion intervals include at least low, medium and high levels. Obtain the credit adjustment strategy corresponding to the target congestion interval, and determine the rate credit and the burst credit based on the credit adjustment strategy.

3. The method according to claim 1, characterized in that, The scheduling and gating are implemented through a two-layer gating mechanism, which includes: The token gate is used to verify the credit limit of each virtual queue according to the two-dimensional credit vector and allow microslices with credit limits that meet preset requirements to pass through. The phase gate receives the micro-slice released by the token gate and, on a preset time grid, decides which micro-slice to release in the end based on arbitration rules, including the debt compensation status; the debt compensation status includes service arrears, last update timestamp, status flag and priority promotion factor.

4. The method according to claim 3, characterized in that, The step of scheduling and gating the microslices in each of the virtual queues on a preset time grid includes: During the scheduling process, when a microslice of any virtual queue is interrupted due to preemption or transfer, the service deficit of the interrupted virtual queue is updated based on the size of the interrupted microslice; and the debt compensation status of the interrupted virtual queue is updated. During subsequent arbitration, the scheduler will raise the release priority of the interrupted virtual queue based on the debt compensation status, so that the interrupted virtual queue can be compensated for the service interruption loss.

5. The method according to claim 3 or 4, characterized in that, The method further includes: The service arrears recorded in the debt compensation status are reduced using a decay strategy to obtain a decayed service arrears; the decay strategy is one of a time-varying decay strategy, a decay function strategy, and an event-related decay strategy. The debt compensation status of each virtual queue is updated based on the attenuation service deficit, so that the scheduling priority of each virtual queue is determined based on the updated debt compensation status.

6. The method according to claim 3, characterized in that, The step of scheduling and gating the microslices in each of the virtual queues on a preset time grid also includes: At the end of a time window, the actual total number of microslices released for each virtual queue is compared with the allocated rate credit. If the actual total number of microslices released is lower than the allocated rate credit limit, the corresponding virtual queue will be determined as a virtual queue with unused credit that needs to be adjusted. Unused credit in the virtual queue to be adjusted is carried over to the next time window at a preset ratio to increase the sudden credit limit of the virtual queue to be adjusted in the next time window.

7. The method according to claim 1, characterized in that, The method further includes: Monitor the congestion intensity reflected by the telemetry data from the system; When the congestion intensity triggers the high-risk threshold, the corresponding virtual queue is identified as a high-risk virtual queue. The high-risk virtual queue is moved into a preset isolation channel, and the channel status is updated; Within the isolation channel, increase the burst credit limit of the high-risk virtual queue and / or lengthen the minimum gate release interval of the high-risk virtual queue; Based on the dynamic changes in the congestion intensity, the sudden credit limit or the minimum gate release interval of the high-risk virtual queue is adjusted linearly or segmentally within a preset range.

8. The method according to claim 7, characterized in that, The method further includes: The fusion result is obtained by fusing the two-dimensional credit vector, channel status, and NVMe controller queue and namespace statistics obtained from system telemetry data; The fusion results are mapped to the commit interval and concurrency depth limit for each namespace to generate an aligned I / O scheduling plan that matches the service tach of the target device. Based on the aligned I / O scheduling plan, a gating and limiting instruction is generated, and the gating and limiting instruction is written back to the scheduling and gating release mechanism.

9. A fine-grained QoS isolation device for NVMe-oF storage, characterized in that, include: The acquisition module is used to acquire system telemetry data and multi-tenant I / O request streams in multi-tenant scenarios; The micro-slice processing module is used to perform micro-slice processing on the multi-tenant I / O request stream and import the resulting micro-slices into the corresponding virtual queues; The credit assessment module is used to determine a two-dimensional credit vector for controlling each of the virtual queues based on the system telemetry data; the two-dimensional credit vector includes rate credit and burst credit. The scheduling module is used to schedule and gate the micro-slices in each virtual queue based on the two-dimensional credit vector of each virtual queue on a preset time grid. The microslicing processing module is also used for: Identify the semantic boundaries existing in the multi-tenant I / O request stream, including the boundaries between NVMe commands and data segments, storage page alignment boundaries, and namespace metadata boundaries; Without compromising the semantic integrity of the request, the identified semantic boundaries are used as the first priority segmentation points to segment the multi-tenant I / O request stream, resulting in micro-slices. When the semantic boundaries are sparse, a nearest truncation strategy is used to determine the second priority segmentation point to segment the multi-tenant I / O request stream, resulting in micro-slices.

Images