A method for monitoring and adaptive adjustment of internal delay in a non-volatile memory controller

Abstract

The application discloses a non-volatile storage controller internal delay monitoring and self-adaptive adjustment method, and belongs to the technical field of non-volatile storage. The method comprises the following steps: collecting total delay of a command and sub-node delay of a NAND layer, and counting proportion and proportion change rate of each delay level under overall dimension and tenant-command type dimension according to time slicing; based on the proportion and change rate of the total delay and the sub-delay, a delay abnormal bottleneck level is determined, a specific root cause is locked by combining a controller state snapshot generated through a preset delay determination condition, self-adaptive optimization such as tenant bandwidth scheduling, background task control and management, data block health management is executed in layers, and a hierarchical host reporting mechanism is configured. The application can realize early prediction and accurate root cause positioning of delay abnormalities, and effectively improve the IO performance stability and service quality guarantee capability of a storage system in a multi-tenant scenario.

Description

Technical Field

[0001] This invention relates to the field of non-volatile memory technology, and in particular to a method for latency monitoring and adaptive adjustment within a non-volatile memory controller. Background Technology

[0002] With the widespread application of enterprise-grade solid-state storage, scenarios of multi-tenant shared storage resources are becoming increasingly common. The stability, observability, and fine-grained control of IO latency have become core indicators for measuring the performance of non-volatile memory controllers.

[0003] Most existing storage latency monitoring solutions only collect macroscopic statistics of total end-to-end latency, failing to break down the latency contributions of the front-end command scheduling layer and the back-end NAND access layer, making it difficult to accurately pinpoint the specific path level of latency anomalies. Most monitoring mechanisms trigger alarms based solely on static latency thresholds, lacking continuous quantitative assessment of latency degradation trends. They are often only discovered when anomalies have already impacted business operations, failing to enable proactive intervention.

[0004] Meanwhile, existing adaptive performance tuning mostly adopts a globally unified scheduling strategy, which cannot combine the root cause level of latency anomalies to perform differentiated optimization, resulting in insufficient tuning accuracy. In multi-tenant scenarios, it is also difficult to achieve fine-grained latency statistics and targeted resource scheduling at the tenant level and command type level, making it difficult to fully guarantee the service quality of core businesses and meet the compliance requirements of enterprise-level storage for standardized latency distribution output. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a method for monitoring and adaptively adjusting latency within a non-volatile memory controller.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] A method for latency monitoring and adaptive adjustment within a non-volatile memory controller includes:

[0008] Receive standardized or custom configuration commands and configure monitoring parameters, including: monitoring trigger mode, tenant identifier, command type, duration and number of time slices, and delay interval division rules;

[0009] The system monitors commands based on a monitoring trigger mode, calculates the total command latency and child node latency of each command, and determines the total latency level and child node latency level of each command based on latency interval division rules. Based on the duration and number of time slices, within each consecutive time slice, the system simultaneously counts the number of commands with each tenant identifier and command type, considering both the total latency level and child node latency level. Based on the number of commands, the system calculates the percentage of total command latency and the percentage of command child node latency. Finally, it compares the percentage of total command latency and the percentage of command child node latency between the current time slice and the previous adjacent time slice to obtain the rate of change of the percentage of total command latency and the rate of change of the percentage of command child node latency.

[0010] Based on the percentage of total command latency, the rate of change of the percentage of total command latency, or the total command latency of a single command, an early warning signal is generated through preset latency judgment conditions; based on the early warning signal, a snapshot of the non-volatile memory controller status is generated.

[0011] Based on the percentage of total command latency, the percentage of command sub-node latency, the rate of change of the percentage of total command latency, the rate of change of the percentage of command sub-node latency, and the state snapshot of the non-volatile memory controller, the root cause of latency anomalies is determined, and the non-volatile memory controller is adaptively adjusted.

[0012] As a preferred embodiment of the present invention, the monitoring triggering mode includes:

[0013] Based on the OCP specification, monitoring can be started within a preset trigger time period or at regular intervals according to a custom monitoring time period preset by the firmware.

[0014] As a preferred embodiment of the present invention, the step of calculating the total command delay and child node delay of the command includes:

[0015] When a command begins execution, the corresponding tenant identifier and command type are parsed, a command tag is assigned to the command, and the total command start timestamp when the command arrives at the non-volatile storage controller interface is recorded. The tenant identifier, command type, and command tag are then combined and written into a preset command record table. When a command is sent to the NAND controller, the start timestamp of the command's child nodes is recorded, and based on the command tag, the start timestamp of the command's child nodes is written into the command record table.

[0016] When a command is completed from the NAND controller, record the end timestamp of the command's child nodes; when the command is completed and the host is notified that the command is completed, record the total command end timestamp.

[0017] Using the command tag of the command as an index, query the command record table to obtain the start timestamp and the start timestamp of the child nodes; subtract the total command start timestamp from the total command end timestamp to obtain the total command delay; subtract the child node start timestamp from the child node end timestamp of the command to obtain the child node delay of the command.

[0018] As a preferred embodiment of the present invention, the step of determining the total delay level of the command and the delay level of the child nodes based on the delay interval division rule includes at least the following:

[0019] When the total delay of a command does not exceed the first preset total delay threshold, the total delay level of the command is confirmed as a low-latency command.

[0020] When the total delay of a command exceeds the first preset total delay threshold but does not exceed the second preset total delay threshold, the total delay level of the command is confirmed as a medium delay command.

[0021] When the total delay of a command exceeds the second preset total delay threshold, the total delay level of the command is confirmed as a high-delay command.

[0022] When the delay of a child node does not exceed the first preset child node delay threshold, the delay level of the child node of the command is confirmed as a low-delay command.

[0023] When the delay of a child node exceeds the first preset child node delay threshold but does not exceed the second preset child node delay threshold, the delay level of the child node of the command is confirmed to be a medium delay command.

[0024] When the delay of a child node exceeds the second preset child node delay threshold, the delay level of the child node of the command is confirmed as a high delay command.

[0025] As a preferred embodiment of the present invention, the method of parallelly counting the number of commands for each tenant identifier and the total latency level and child node latency level under any command type; and calculating the total latency ratio and child node latency ratio of commands based on the number of commands, includes:

[0026] The total command latency percentage includes the overall total latency percentage under different total latency levels and the total latency percentage of a single category;

[0027] The overall total latency percentage is the ratio of the total number of commands of all tenant identifiers and all command types corresponding to each total latency level within a single time slice to the total number of commands within that time slice.

[0028] The single-category total latency percentage is the ratio of the number of commands of any tenant identifier and any command type corresponding to each total latency level within a single time slice to the total number of commands of that tenant identifier and any command type within that time slice.

[0029] The command sub-node latency percentage includes the overall sub-node latency percentage under different sub-node latency levels and the latency percentage of a single type of sub-node;

[0030] The overall sub-node latency ratio is the ratio of the total number of commands of all tenant identifiers and all command types corresponding to the latency level of each sub-node within a single time slice to the total number of commands within that time slice.

[0031] The single-type sub-node latency ratio is the ratio of the number of commands of any tenant identifier and any command type corresponding to the latency level of each sub-node within a single time slice to the total number of commands of that tenant identifier and any command type within that time slice.

[0032] As a preferred embodiment of the present invention, obtaining the change rate of the total command delay ratio and the change rate of the command sub-node delay ratio includes:

[0033] The percentage change in total command latency includes the percentage change in overall total latency and the percentage change in total latency for a single category;

[0034] The difference between the total total latency percentage of each total latency level in the current time slice and the corresponding total total latency percentage in the previous adjacent time slice is divided by the duration of the current time slice to obtain the corresponding rate of change of the total total latency percentage.

[0035] The difference between the percentage of total latency for each tenant identifier and command type under each total latency level within the current time slice and the percentage of total latency for the corresponding single category in the previous adjacent time slice is divided by the duration of the current time slice to obtain the corresponding percentage change rate of total latency for the single category.

[0036] The change rate of the delay ratio of the command sub-nodes includes the change rate of the overall delay ratio of sub-nodes and the change rate of the delay ratio of a single type of sub-node;

[0037] The difference between the overall child node latency percentage under each child node latency level in the current time slice and the corresponding overall child node latency percentage in the previous adjacent time slice is divided by the duration of the current time slice to obtain the corresponding rate of change of the overall child node latency percentage.

[0038] The difference between the single-type child node latency percentage under each child node latency level for any tenant identifier and any command type within the current time slice and the corresponding single-type child node latency percentage in the previous adjacent time slice is divided by the duration of the current time slice to obtain the corresponding single-type child node latency percentage change rate.

[0039] As a preferred embodiment of the present invention, the preset delay determination condition includes at least:

[0040] If, within a single time slice, the total delay of any command exceeds the preset single command delay warning threshold, or if, within a single time slice, the proportion of the total delay corresponding to the preset total delay level exceeds the preset total proportion warning threshold, or if, within a consecutive preset number of time slices, the rate of change of the proportion of the total delay corresponding to commands with a high delay level exceeds the preset deterioration rate warning threshold, then a warning signal is generated.

[0041] Based on the warning signal, a snapshot of the non-volatile memory controller's state is generated according to the internal state information of the non-volatile memory controller. The state information includes, but is not limited to, the running state of the state machine inside the non-volatile memory controller, the request queue depth, the completion queue depth, error status information, the number of error correction decoding iterations, and the background task type and execution status.

[0042] As a preferred embodiment of the present invention, the step of determining the root cause of the delay anomaly and adaptively adjusting the non-volatile memory controller includes:

[0043] a. When, within a consecutive preset number of time slices, the overall total latency percentage of commands with a high latency level exceeds the preset overall total latency percentage abnormal threshold or the corresponding overall total latency percentage change rate exceeds the preset overall total deterioration rate threshold, and the overall latency percentage of corresponding child nodes with a high latency level does not exceed the preset overall child latency percentage abnormal threshold or the corresponding overall child latency percentage change rate does not exceed the preset overall child deterioration rate threshold, then the latency abnormality bottleneck level is confirmed to be at the front-end scheduling layer. The abnormal tenant is determined by combining the single-type total latency percentage of different tenants, the actual load and the allocated bandwidth percentage threshold, and the command scheduling weight or scheduling priority of the abnormal tenant is adjusted to reduce the bandwidth quota.

[0044] b. If, within a consecutive preset number of time slices, the overall total latency percentage of commands with a high latency level exceeds a preset overall total latency percentage abnormal threshold or the corresponding overall total latency percentage change rate exceeds a preset overall total degradation rate threshold, and the overall child node latency percentage of corresponding child node commands with a high latency level exceeds a preset overall child latency percentage abnormal threshold and the corresponding overall child node latency percentage change rate exceeds a preset overall child degradation rate threshold, then the latency abnormality bottleneck level is determined to be at the backend NAND control layer. If the non-volatile storage controller status snapshot shows that the background is currently performing garbage collection or cold data migration tasks, then the background task priority is adjusted to prioritize responding to foreground IO requests. If the non-volatile storage controller status snapshot shows that the number of error correction decoding iterations of data read from NAND exceeds a preset abnormal threshold, then the data in the corresponding physical storage block is moved to an available physical storage block with an erase count lower than a preset erase count threshold and a bit error rate within a preset normal range.

[0045] In a preferred embodiment of the present invention, the total command delay percentage further includes the total interval distribution percentage, and the total command delay percentage change rate further includes the total interval distribution percentage change rate; the total interval distribution percentage is a set of the overall total delay percentages corresponding to all total delay levels arranged in the order of delay intervals, and the total interval distribution percentage change rate is a set of the overall total delay percentage change rates corresponding to each total delay level between adjacent time slices; the total interval distribution percentage and the total interval distribution percentage change rate are used to generate a delay distribution histogram, conduct delay distribution trend analysis, and verify the adaptive adjustment effect;

[0046] The command sub-node latency percentage also includes the sub-interval distribution percentage, and the command sub-node latency percentage change rate also includes the sub-interval distribution percentage change rate. The sub-interval distribution percentage is a set of the overall sub-node latency percentages corresponding to all sub-node latency levels arranged in the order of latency intervals, and the sub-interval distribution percentage change rate is a set of the overall sub-node latency percentage change rates corresponding to each sub-node latency level between adjacent time slices. The sub-interval distribution percentage and the sub-interval distribution percentage change rate are used to generate a NAND path latency distribution histogram, conduct NAND path latency distribution trend analysis, verify the adaptive adjustment effect of the NAND layer, and assist in locating the bottleneck level of latency anomalies.

[0047] As a preferred embodiment of the present invention, when at least one preset reporting trigger condition is met, the delay statistics log for the corresponding time slice is recorded and reported to the host; the preset reporting trigger conditions include:

[0048] Within any time slice, the total delay of a single command exceeds the preset total command reporting threshold;

[0049] Within any time slice, the overall total latency percentage of commands with a total latency level of high latency exceeds the preset overall high latency percentage reporting threshold;

[0050] Within a consecutive preset number of time slices, the overall total latency percentage of commands with a high latency level exceeds the preset consecutive latency percentage reporting threshold.

[0051] The latency statistics log includes at least the percentage of total latency for each tenant identifier and command type, the rate of change of the percentage of total latency for each type, the percentage of latency for each sub-node, and the rate of change of the percentage of latency for each sub-node within the corresponding time slice.

[0052] The beneficial effects of this invention are as follows: First, by synchronously collecting total command latency and child node latency, the latency bottleneck levels of the front-end scheduling layer and the back-end NAND control layer can be accurately distinguished, solving the problem that existing solutions can only count end-to-end total latency and cannot locate the root cause path of anomalies, providing an accurate basis for targeted performance tuning. Second, by using a time slicing mechanism to count the proportion and rate of change of each latency level, it breaks through the limitations of static threshold alarms, can capture the dynamic trend of continuous latency degradation, and realize early prediction and intervention of latency anomalies, effectively improving the lag of traditional alarm mechanisms. Third, it supports parallel statistics of multi-dimensional latency data by tenant identifier and command type, which can accurately locate abnormal tenants and abnormal command types in scenarios where multiple tenants share storage resources, realizing tenant-level fine-grained performance control and fully ensuring the service quality of core businesses. Finally, by combining controller state snapshots to lock down specific anomaly root causes and matching differentiated adaptive adjustment strategies, it replaces the globally unified coarse scheduling, improves the accuracy of performance adjustment, and minimizes the impact on normal businesses while suppressing latency anomalies, thereby improving the overall IO performance stability of the storage system. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart of the method in an embodiment of the present invention; Figure 2 Here is an example of a histogram showing the percentage of total latency corresponding to different total latency levels in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the changing trend and deterioration rate of the overall total latency percentage for medium and high latency commands under continuous time slices in this embodiment of the invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0055] like Figures 1 to 3 As shown, this is an embodiment of the present invention, which provides a method for latency monitoring and adaptive adjustment within a non-volatile memory controller, including:

[0056] S1. Receive standardized configuration instructions or custom configuration instructions, and configure monitoring parameters, including: monitoring trigger mode, tenant identifier, command type, duration and number of time slices, and delay interval division rules.

[0057] Specifically, the configuration management module supports receiving two types of input: standardized configuration commands and custom configuration commands. The standardized configuration commands conform to the command format of the OCP storage device telemetry specification and are issued by the host system through the standard management interface. The custom configuration commands are generated by the controller firmware according to preset business policies and are used to adapt to the monitoring needs of specific scenarios.

[0058] After completing the command parsing, the configuration management module configures the following core monitoring parameters in sequence:

[0059] 1. Monitoring trigger mode, used to set the start and stop rules of the delay monitoring function, including two types: standard trigger mode and custom timed trigger mode;

[0060] 2. Tenant Identifier: Used as the tenant identity identifier based on the namespace identifier in the NVMe protocol. The default tenant identifiers included in the statistics are NS0, NS1, and NS2. It can also be configured to a full namespace statistics mode, covering the IO commands of all tenants under the controller.

[0061] 3. Command type: The default command types included in the statistics are NVMe read commands, NVMe write commands, and TRIM commands. It supports separate statistics for read commands, separate statistics for write commands, or combined statistics for read and write commands. The default is to use the combined statistics mode for read and write commands.

[0062] 4. The maximum number of supported slices is determined by the depth of the hardware-reserved statistical register array. This statistical register array uses a shift-latch architecture to form a sliding monitoring time window that can continuously shift forward over time. Accuracy levels are switched by adjusting the duration of a single time slice. In this embodiment, the preset standard level is: a single time slice duration of 100 milliseconds, enabling all 16 consecutive statistical time slices, forming a sliding monitoring time window with a total duration of 1.6 seconds. This parameter supports dynamic configuration of the level according to monitoring accuracy and overhead requirements, for example:

[0063] In high-precision scenarios, the 16 time slices remain unchanged, but the duration of each time slice is configured to 10 milliseconds, corresponding to a sliding monitoring window duration of 160 milliseconds. This setting only improves the granularity of time statistics without increasing hardware storage resource overhead, making it suitable for fine-grained capture scenarios with short-term peak latency.

[0064] In low-overhead scenarios, the single time slice duration is configured to 200 milliseconds, and only 4 time slices are enabled at the same time, corresponding to a sliding monitoring window duration of 800 milliseconds. This setting reduces hardware resource consumption and power consumption by reducing the number of enabled counter array groups, making it suitable for long-term background inspection monitoring scenarios.

[0065] At the end of each time slice's timing, the hardware automatically shifts the historical time slice statistics stored in each register level one position towards the older data, with the full-dimensional statistics corresponding to the earliest time slice being removed and overwritten. Subsequently, the statistics of the currently calculated time slice are latched into the latest level of the statistics register array. Through this mechanism, the monitoring window always covers the N consecutive time slices closest to the current moment, continuously sliding forward over time, maintaining the window's real-time performance and continuity without firmware intervention.

[0066] 5. Delay interval division rules: Configure multi-level delay thresholds corresponding to the total delay of the command and the delay of the child node, which are used to divide different delay levels. The corresponding thresholds can be dynamically adjusted according to the characteristics of the storage medium and the performance requirements of the business.

[0067] In this embodiment, the monitoring triggering mode is implemented as follows:

[0068] The first type is the triggering mode based on the OCP specification, which follows the trigger window mechanism defined by the OCP storage device telemetry standard. In this embodiment, the preset triggering time period is the peak business window from 10:00 to 17:00 every day. The host issues configuration parameters through the OCP standard management interface. When the trigger start time is reached, the configuration management module automatically starts the full latency statistics and monitoring process, and automatically stops monitoring and generates standardized telemetry logs when the trigger end time is reached.

[0069] The second type is a custom timed trigger mode, in which the controller firmware presets the monitoring cycle according to the business load pattern. In this embodiment, the high latency anomaly monitoring is preset to be activated for the first 5 minutes of each hour, while only the single command extreme latency alarm function is retained during the remaining time periods. This mode can reduce the occupation of the controller's computing power and storage resources by the monitoring function while ensuring the anomaly detection capability, and achieve a dynamic balance between monitoring overhead and business performance.

[0070] S2. Monitor commands based on the monitoring trigger mode, calculate the total command latency and child node latency of the command, and determine the total latency level and child node latency level of the command based on the latency interval division rules; based on the duration and number of time slices, in each consecutive time slice, count the number of commands with total latency level and child node latency level under each tenant identifier and any command type in parallel; and calculate the total latency ratio and child node latency ratio of the command based on the number of commands; compare the total latency ratio and child node latency ratio of the current time slice with the previous adjacent time slice to obtain the change rate of the total latency ratio and the change rate of the child node latency ratio.

[0071] Specifically, when a command issued by the host enters the non-volatile storage controller interface and begins execution, the command parsing module first parses the command frame to extract the tenant identifier and command type corresponding to the command; then, it assigns a globally unique command tag to the command and reads the current count value of the high-precision timer inside the controller, recording it as the total command start timestamp; the command parsing module writes the tenant identifier, command type, command tag, and total command start timestamp into a preset command record table; the command record table uses the command tag as the primary key index and is used to store the attribute information and timing information of a single command throughout its entire lifecycle.

[0072] When the command is sent to the NAND controller after being processed by the front-end scheduling and address mapping, the NAND control interface module reads the current count value of the current high-precision timer and records it as the start timestamp of the child node of the command; and uses the command tag of the command as an index to update the start timestamp of the child node and write it to the corresponding entry in the command record table.

[0073] When the command is executed on the NAND controller side and returns the execution result, the NAND control interface module reads the current high-precision timer count value as the child node end timestamp; using the command tag as an index, it reads the corresponding child node start timestamp from the command record table, subtracts the child node start timestamp from the child node end timestamp, and calculates the child node delay of the command in real time. The child node delay calculation result is directly output to the delay statistics module and is not stored in the command record table to reduce the read / write overhead of on-chip memory.

[0074] When the command completes all front-end processing and sends a command completion notification to the host, the command completion module reads the current high-precision timer count as the total command end timestamp; using the command tag as an index, it reads the corresponding total command start timestamp from the command record table, subtracts the total command start timestamp from the total command end timestamp, and calculates the total command delay in real time; the total command delay covers the entire path time from entering the non-volatile memory controller interface to completing the host notification. The total command delay calculation result is synchronously output to the delay statistics module, and the entry corresponding to the command tag in the command record table is released for reuse by newly issued commands. The command record table in this embodiment is shown in Table 1:

[0075] Table 1 Command Record Table

[0076]

[0077] In this embodiment, when the total delay of a command does not exceed a first preset total delay threshold, the total delay level of the command is confirmed as a low-latency command; when the total delay of a command exceeds the first preset total delay threshold but does not exceed a second preset total delay threshold, the total delay level of the command is confirmed as a medium-latency command; when the total delay of a command exceeds the second preset total delay threshold, the total delay level of the command is confirmed as a high-latency command. Similarly, when the delay of a child node does not exceed a first preset child node delay threshold, the delay level of the child node of the command is confirmed as a low-latency command; when the delay of a child node exceeds the first preset child node delay threshold but does not exceed a second preset child node delay threshold, the delay level of the child node of the command is confirmed as a medium-latency command; and when the delay of a child node exceeds the second preset child node delay threshold, the delay level of the child node of the command is confirmed as a high-latency command.

[0078] Specifically, the steps of determining the total delay level of the command and the delay level of the child nodes based on the preset delay interval division rules are executed by the delay level determination module inside the non-volatile memory controller. This module is implemented based on hardware comparison circuits and can complete the level determination within a single clock cycle after the command delay calculation is completed, without firmware intervention, thus ensuring the real-time performance of delay statistics.

[0079] In this embodiment, the first preset total latency threshold is 100 microseconds, and the second preset total latency threshold is 500 microseconds. Both are stored in a readable and writable hardware configuration register and can be dynamically adjusted by firmware according to the storage medium type and service level requirements. The first preset child node latency threshold is 80 microseconds, and the second preset child node latency threshold is 400 microseconds. This set of thresholds also supports dynamic firmware configuration and can adapt to the performance characteristics of different processes and types of NAND flash memory media.

[0080] In this embodiment, the total command latency percentage includes the overall total latency percentage under different total latency levels and the single-category total latency percentage; the overall total latency percentage is the ratio of the total number of commands of all tenant identifiers and all command types corresponding to each total latency level within a single time slice to the total number of commands within that time slice; the single-category total latency percentage is the ratio of the number of commands of any tenant identifier and any command type corresponding to each total latency level within a single time slice to the total number of commands of that tenant identifier and any command type within that time slice; the command sub-node latency percentage includes the overall sub-node latency percentage and the single-category sub-node latency percentage under different sub-node latency levels; the overall sub-node latency percentage is the ratio of the total number of commands of all tenant identifiers and all command types corresponding to each sub-node latency level within a single time slice to the total number of commands within that time slice; the single-category sub-node latency percentage is the ratio of the number of commands of any tenant identifier and any command type corresponding to each sub-node latency level within a single time slice to the total number of commands of that tenant identifier and any command type within that time slice.

[0081] Specifically, the statistical calculation of the total command latency ratio and the command sub-node latency ratio is implemented by the latency statistics module inside the non-volatile memory controller based on a hardware parallel counter array. It completes full-dimensional counting and ratio calculation within each consecutive time slice period, and the statistical process does not require firmware intervention.

[0082] In this embodiment, the percentage of total latency corresponding to different total latency levels within each time slice is statistically analyzed in parallel. The statistical results are as follows: Figure 2 As shown. Figure 2 The horizontal axis represents the time slices of continuous acquisition. The percentage of different latency levels represents the overall total latency percentage corresponding to different total latency levels. The percentage of commands with low, medium, and high total latency levels represents the overall total latency percentage of commands with low, medium, and high latency levels. As time slice 0 progresses to time slice 2, the overall total latency percentage of commands with low latency level gradually decreases, while the overall total latency percentage of commands with medium and high latency levels continuously increases, reflecting the basic statistical data of the gradual deterioration of command latency in the storage system.

[0083] Taking a single time slice of 100 milliseconds as an example, if the total number of commands completed within this slice is 1000, and 80 of them fall into the category of high-latency commands, then the overall latency percentage corresponding to high-latency commands is 8%. Taking write commands for tenant identifier NS0 as an example, if there are 200 write commands for tenant NS0 within a single time slice, and 12 of them fall into the category of high-latency commands, then the single-category total latency percentage for tenant NS0 write commands that are high-latency commands is 6%.

[0084] All statistical calculation results are automatically latched and shifted by the hardware at the end of each time slice. The non-volatile memory controller has a multi-level statistical register array that corresponds one-to-one with the preset number of time slices, forming a continuous sliding monitoring time window. When a single time slice ends, the hardware first shifts the historical slice statistical data stored in each level of register one level towards the older data, with the statistical data corresponding to the earliest time slice being shifted out and overwritten. Then, all statistical results that have been settled in the current time slice, including the delay percentage of each dimension and the maximum total delay of a single command, are latched into the latest level of statistical register. This storage mechanism ensures that the statistical register array always stores statistical data from multiple consecutive time slices, with data from adjacent time slices corresponding one-to-one, allowing the rate of change calculation module to directly read data from two adjacent sets of slices to solve for the rate of change. On the other hand, all statistical data can be read by the firmware through the standard register interface for delay distribution trend analysis and delay statistical log reporting.

[0085] In this embodiment, the change rate of the total command latency percentage includes the overall total latency percentage change rate and the change rate of the single-type total latency percentage change rate. The overall total latency percentage change rate is obtained by subtracting the corresponding overall total latency percentage in the previous adjacent time slice from the overall total latency percentage under each total latency level within the current time slice, and then dividing the difference by the duration of the current time slice. Similarly, the change rate of the single-type total latency percentage is obtained by subtracting the corresponding single-type total latency percentage in the previous adjacent time slice from the single-type total latency percentage under each total latency level for any tenant identifier and any command type within the current time slice, and then dividing the difference by the duration of the current time slice. The command child node latency... The time-sharing percentage change rate includes the overall child node latency percentage change rate and the single-type child node latency percentage change rate. The overall child node latency percentage change rate is obtained by subtracting the corresponding overall child node latency percentage in the previous adjacent time slice from the overall child node latency percentage under each child node latency level within the current time slice, and then dividing the difference by the duration of the current time slice. Similarly, the single-type child node latency percentage change rate is obtained by subtracting the corresponding single-type child node latency percentage in the previous adjacent time slice from the single-type child node latency percentage under each child node latency level for any tenant identifier and any command type within the current time slice, and then dividing the difference by the duration of the current time slice.

[0086] Specifically, the calculation of the percentage change rate of total command latency and the percentage change rate of command sub-node latency is performed by the latency change rate statistics module inside the non-volatile memory controller. This module latches the percentage statistics results of adjacent time slices through hardware registers and uses dedicated hardware computing circuits to solve the percentage change per unit time without firmware intervention.

[0087] In this embodiment, taking a 100-millisecond time slice as an example, if the overall total latency percentage corresponding to high-latency commands in the previous adjacent time slice is 3%, and the overall total latency percentage of the corresponding dimension in the current time slice is 5%, then the difference in percentage is 2%, corresponding to a change rate of 2% / 100ms for the overall total latency percentage, meaning that the high-latency command increases by 2 percentage points every 100 milliseconds. Taking the write command of tenant identifier NS0 as an example, if the single-category total latency percentage of high-latency commands in the previous adjacent time slice is 4%, and the single-category total latency percentage of the corresponding dimension in the current time slice is 7%, and the time slice duration is 100 milliseconds, then the corresponding single-category total latency percentage change rate is 3% / 100ms.

[0088] All calculation results are output synchronously at the end of each time slice, and are sent to the early warning judgment module on the one hand, and stored in the on-chip statistical register on the other hand for firmware reading and log reporting.

[0089] S3. Based on the percentage of total command latency, the rate of change of the percentage of total command latency, or the total command latency of a single command, generate an early warning signal through preset latency judgment conditions; based on the early warning signal, generate a snapshot of the non-volatile memory controller status.

[0090] Specifically, the preset delay determination conditions include at least the following three categories. When any one of the determination conditions is met, the warning determination module generates a warning signal:

[0091] 1. Within a single time slice, if the total delay of any command exceeds a preset single-command delay warning threshold, an warning signal is triggered. In this embodiment, the preset single-command delay warning threshold is 2 milliseconds. If the total delay of a host write command is 2.3 milliseconds, the preset delay judgment condition is met, and the warning judgment module immediately generates a warning signal to capture extreme anomalies at the single-command level.

[0092] 2. Within a single time slice, if the proportion of total latency corresponding to a preset total latency level exceeds a preset total latency warning threshold, an early warning signal is triggered. In this embodiment, the preset total latency level is high-latency commands, and the preset total latency warning threshold is 10%. Within a single time slice of 100 milliseconds, if the proportion of total latency corresponding to high-latency commands is 12%, the judgment condition is met, and an early warning signal is generated to identify global latency degradation within a single period.

[0093] 3. Within a consecutive preset number of time slices, if the rate of change of the overall total latency percentage corresponding to commands with a high latency level exceeds a preset deterioration rate warning threshold, an early warning signal is triggered. In this embodiment, the consecutive preset number is 3, and the preset deterioration rate warning threshold is 2% / 100ms. If, within three consecutive time slices, the rate of change of the overall total latency percentage corresponding to commands with a high latency level is 2.1% / 100ms, 2.5% / 100ms, and 2.3% / 100ms respectively, and all three cycles exceed the deterioration rate warning threshold, then it is determined that the latency is in a continuous deterioration trend, thus satisfying the judgment condition and generating an early warning signal, achieving advance prediction of latency anomalies.

[0094] In this embodiment, one optional approach is to preset the total latency level as medium or high latency commands as the judgment level; such as Figure 3 As shown in the figure, the purple dashed line represents the set total latency percentage threshold. The overall total latency percentage for medium-high latency levels is the overall total latency percentage corresponding to commands with medium or high latency levels. When the overall total latency percentage exceeds this line within a time slice, an early warning is triggered. The green dashed line represents the deterioration rate trend of the overall total latency percentage for commands with medium or high latency levels continuously increasing within consecutive time slices. When the rate of change of the overall total latency percentage corresponding to commands with medium or high latency levels exceeds the preset medium-high deterioration rate threshold within consecutive time slices, it is determined that the system latency is continuously deteriorating, and an early warning signal is generated. In this embodiment, the preset medium-high deterioration rate threshold is 2% / 100ms.

[0095] Within the same clock cycle or a preset number of cycles after the warning signal is generated, the snapshot generation module synchronously latches the real-time state information inside the non-volatile memory controller, generating a non-volatile memory controller state snapshot. This non-volatile memory controller state snapshot is a panoramic slice of the controller's operating state at the moment the warning is triggered; the data is statically latched and unaffected by subsequent operating states. The real-time state information includes the state machine operating state, request queue depth, completion queue depth, error state information, error correction decoding iteration count, background task type, and execution status.

[0096] The state machine's running state includes the current running stage and working mode of the front-end command scheduling state machine, the flash memory conversion layer processing state machine, and the NAND interface control state machine, which are used to determine whether there are problems such as abnormal state machine suspension or scheduling process deadlock.

[0097] The request queue depth and completion queue depth record the number of pending commands in the host-side submission queue and the number of pending commands in the controller-side completion queue, respectively. These are used to assess the congestion level of the IO link and determine whether the latency anomaly is caused by command backlog.

[0098] Error status information includes hardware register information such as NAND read / write error identifiers, media verification error counts, and bus transmission error status, which are used to troubleshoot media failures and data transmission anomalies.

[0099] The error correction decoding iteration count records the actual number of error correction decoding iterations corresponding to the current NAND read operation, which is used to indirectly evaluate the health status of the flash memory physical block and the original bit error rate level.

[0100] The background task type and execution status record records the currently running background maintenance task categories, including garbage collection, cold data migration, full wear leveling, etc., as well as the corresponding task execution priority, concurrent channel count, and other running parameters, which are used to determine whether background tasks are preempting foreground IO channels and computing resources.

[0101] The generated non-volatile memory controller state snapshot is stored in the on-chip dedicated cache area, which is used for subsequent adaptive adjustment modules to read; on the other hand, it can be reported to the host along with the latency statistics log to support fault backtracking and performance analysis on the operation and maintenance side.

[0102] At the same moment the warning signal is effectively generated, the hardware automatically terminates the automatic shift update of the current sliding monitoring time window, freezes the full-dimensional count data and percentage calculation results in the statistical register array at each level, and suspends the delay level count update of new commands, that is, stops the current real-time delay monitoring process, and completely preserves the statistical scene and controller status snapshot at the moment the warning is triggered, so as to avoid subsequent running data from overwriting the abnormal scene and ensure that the data source of firmware analysis is accurate and effective.

[0103] After monitoring is paused, the controller firmware intervenes to perform anomaly analysis. Once the firmware completes all analysis and handling processes, it sends a monitoring recovery command to the hardware module. The hardware clears the counter registers of each dimension, resets the storage units at each level of the sliding monitoring time window, restarts the full-process delayed monitoring and statistical update, and restores normal monitoring operation.

[0104] S4. Based on the percentage of total command latency, the percentage of command sub-node latency, the rate of change of total command latency, the rate of change of command sub-node latency, and the snapshot of the non-volatile memory controller status, determine the root cause of latency anomalies and adaptively adjust the non-volatile memory controller.

[0105] Specifically, the root cause determination module and the adaptive adjustment module inside the non-volatile memory controller work together to divide the abnormal bottleneck levels based on the ratio and rate of change of the total command latency and the child node latency. Combined with the state snapshot of the non-volatile memory controller, the specific root cause is identified, and a differentiated adaptive optimization strategy is executed to achieve precise intervention and dynamic performance optimization of latency anomalies.

[0106] When, within a consecutive preset number of time slices, the proportion of total latency corresponding to commands with a high latency level exceeds a preset overall total latency abnormal threshold, or the rate of change of the corresponding overall total latency proportion exceeds a preset overall total degradation rate threshold, and the proportion of overall sub-node latency corresponding to commands with a high latency level does not exceed a preset overall sub-proportion abnormal threshold, and the rate of change of the corresponding overall sub-node latency proportion does not exceed a preset overall sub-degradation rate threshold, the root cause determination module confirms that the bottleneck level of the latency anomaly is located in the front-end scheduling layer. That is, the latency degradation is caused by congestion in the front-end links such as command scheduling queuing and flash memory translation layer address mapping processing, and is unrelated to the back-end NAND media access performance.

[0107] In this embodiment, the preset number of consecutive time slices is 3, and the duration of a single time slice is 100 milliseconds. The preset overall total percentage abnormal threshold is 10%, and the preset overall total deterioration rate threshold is 2% / 100ms. The preset overall sub-percentage abnormal threshold is 8%, and the preset overall sub-deterioration rate threshold is 1.5% / 100ms. For example, within three consecutive time slices, the overall total latency percentages corresponding to commands with high latency are 11%, 12%, and 13%, respectively, all exceeding the overall total percentage abnormal threshold. The overall sub-node latency percentages of commands with high latency in the corresponding period are 5%, 6%, and 5.5%, respectively, all not exceeding the overall sub-percentage abnormal threshold. Furthermore, the rate of change of the overall sub-node latency percentage is lower than the overall sub-deterioration rate threshold, indicating that the bottleneck is located in the front-end scheduling layer.

[0108] After identifying the bottleneck in the front-end scheduling layer, the root cause determination module determines abnormal tenants by combining the percentage of total latency for each tenant's single category, actual load, and allocated bandwidth percentage threshold. In this embodiment, the preset bandwidth percentage threshold for a single tenant is 30%. If the percentage of total latency for a single category of commands corresponding to the tenant identifier NS0, which is high-latency commands, reaches 18%, and its actual IO load accounts for 35% of the controller's total bandwidth, exceeding the allocated bandwidth quota, then tenant NS0 is determined to be an abnormal tenant, whose excessive use of front-end scheduling resources leads to congestion in the overall scheduling queue.

[0109] After locating the abnormal tenant, the adaptive adjustment module adjusts the command scheduling weight or priority of the abnormal tenant, reducing its bandwidth quota. For example, the command scheduling weight of tenant NS0 is reduced by 30% from the default value, the scheduling priority of the corresponding submission queue is reduced, and the maximum IO bandwidth is limited to the allocated threshold range, thereby alleviating the congestion of the front-end scheduling queue and gradually reducing the overall system latency.

[0110] Specifically, when, within a consecutive preset number of time slices, the overall total latency percentage corresponding to commands with a high latency level exceeds a preset overall total latency percentage abnormal threshold, or the corresponding overall total latency percentage change rate exceeds a preset overall total degradation rate threshold, and the overall child node latency percentage corresponding to commands with a high latency level exceeds a preset overall child latency percentage abnormal threshold and the corresponding overall child node latency percentage change rate exceeds a preset overall child degradation rate threshold, the root cause determination module determines that the bottleneck level of the latency anomaly is located in the backend NAND control layer, that is, the latency degradation is caused by the performance degradation of the NAND access path.

[0111] In this embodiment, if the overall total latency percentage of commands with a high latency level changes by 2.2% / 100ms within three consecutive time slices, exceeding the overall total degradation rate threshold; and the overall child node latency percentage of commands with a high latency level changes by 2.0% / 100ms within the synchronization period, exceeding the overall child degradation rate threshold, and both show a synchronous degradation trend, it can be confirmed that the latency anomaly is caused by a decrease in the performance of the backend NAND access path.

[0112] After identifying the bottleneck in the backend NAND control layer, the root cause determination module, in conjunction with a snapshot of the non-volatile memory controller's state, further pinpoints the specific root cause and executes corresponding adjustment strategies, specifically in two scenarios:

[0113] The first category is background task resource preemption scenarios. If the non-volatile storage controller status snapshot shows that the background is currently executing garbage collection or cold data migration tasks, the root cause of the latency anomaly is determined to be the background maintenance task preempting NAND channel bandwidth and arbitration resources. In this case, the adaptive adjustment module adjusts the priority of background tasks, prioritizing responses to foreground IO requests; for example, lowering the NAND channel access arbitration priority of garbage collection tasks, increasing the arbitration weight of foreground IO read and write requests to the highest level, limiting the maximum concurrent NAND channel count of garbage collection tasks, suspending non-urgent cold data centralized migration tasks, and releasing backend bandwidth resources to prioritize foreground business IO.

[0114] The second category is the scenario of media health degradation. If the non-volatile memory controller status snapshot shows that the number of error correction decoding iterations of the data read from NAND exceeds a preset abnormal threshold, it is determined that the media health of the corresponding physical storage block has decreased and the original bit error rate has increased. In this embodiment, the preset abnormal threshold is 8 times. If the number of error correction decoding iterations during the reading of a physical storage block reaches 11 times, exceeding the preset abnormal threshold, it is confirmed that the reliability of the physical storage block has deteriorated. At this time, the adaptive adjustment module moves the data in the corresponding physical storage block to an available physical storage block with an erase count lower than the preset erase count threshold and a bit error rate within the preset normal range. In this embodiment, the preset erase count threshold is 3000 times, and the preset normal range is less than 1000 times. Select idle physical storage blocks with an erase count of 800 and a bit error rate within the normal range to complete the data migration. After the data migration is completed, the original physical storage blocks with high bit error rates are included in the wear leveling queue to avoid continuous high latency in subsequent read and write operations.

[0115] After the adaptive adjustment strategy is implemented, real-time monitoring is continuously performed on the overall total latency percentage, the rate of change of the overall total latency percentage, the overall child node latency percentage, and the rate of change of the overall child node latency percentage within subsequent consecutive time slices. If, within five consecutive time slices, the overall total latency percentage of commands with a high latency level is less than 10% and the absolute value of the rate of change of the overall total latency percentage is less than 2% / 100ms, and the overall child node latency percentage of commands with a high latency level is less than 8% and the absolute value of the rate of change of the overall child node latency percentage is less than 1.5% / 100ms, then the system latency anomaly is determined to have been eliminated. The controller gradually recalls the command scheduling weights of the abnormal tenants according to the gradient recovery logic. In this embodiment, every two time slices, 10% of the command scheduling weights, background task access priorities, and other adjustment parameters are recalled to the factory default configuration to avoid secondary fluctuations in system latency caused by a one-time parameter recovery.

[0116] In this embodiment, the total command latency percentage also includes the total interval distribution percentage, and the total command latency percentage change rate also includes the total interval distribution percentage change rate; the total interval distribution percentage is the set of the overall total latency percentages corresponding to all total latency levels arranged in the order of latency intervals, and the total interval distribution percentage change rate is the set of the overall total latency percentage change rates corresponding to each total latency level between adjacent time slices; the total interval distribution percentage and the total interval distribution percentage change rate are used to generate latency distribution histograms, conduct latency distribution trend analysis, and verify the adaptive adjustment effect.

[0117] Specifically, within a single 100-millisecond time slice, if the overall total latency percentage for low-latency commands is 85%, for medium-latency commands it is 10%, and for high-latency commands it is 5%, then the total interval distribution percentages for that slice are arranged in order as [85%, 10%, 5%]. The total interval distribution percentages and the rate of change of the total interval distribution percentages are used to generate a latency distribution histogram, which visually presents the overall distribution structure of the system's end-to-end latency. Latency distribution trend analysis is conducted to track the migration patterns of the proportions of each latency level and identify the long-term degradation trend of the latency distribution. The adaptive adjustment effect is verified by comparing the distribution percentage data before and after the adjustment strategy is implemented to quantitatively evaluate the degree of optimization.

[0118] The command sub-node latency percentage also includes the sub-interval distribution percentage, and the command sub-node latency percentage change rate also includes the sub-interval distribution percentage change rate. The sub-interval distribution percentage is a set of the overall sub-node latency percentages corresponding to all sub-node latency levels arranged in latency interval order, fully presenting the latency distribution status of the NAND access path within a single time slice. The sub-interval distribution percentage change rate is a set of the overall sub-node latency percentage change rates corresponding to each sub-node latency level between adjacent time slices, used to characterize the dynamic change rate of the backend NAND path latency distribution. The uses of the sub-interval distribution percentage and the sub-interval distribution percentage change rate are: first, to generate a NAND path latency distribution histogram, intuitively displaying the latency structure of backend media access; second, to conduct NAND path latency distribution trend analysis and monitor the long-term change trend of flash memory media performance; and third, to verify... The system is designed to verify the adaptive adjustment effect of the NAND layer and quantify the performance optimization benefits of strategies such as background task scheduling and data migration. Fourthly, it assists in locating bottleneck levels for latency anomalies. Through comprehensive comparative analysis of the total interval distribution ratio and the rate of change of the total interval distribution ratio, the bottleneck level is determined hierarchically: if only the total interval distribution ratio and the rate of change of the total interval distribution ratio show a deteriorating trend, while the corresponding sub-interval distribution ratio and the rate of change of the sub-interval distribution ratio remain stable without a significant increase, then the bottleneck level for latency anomalies is determined to be located in the front-end scheduling layer, and the latency deterioration is caused by front-end command queuing and resource scheduling congestion; if the total interval distribution ratio and the rate of change of the total interval distribution ratio and the rate of change of the sub-interval distribution ratio and the rate of change of the sub-interval distribution ratio both deteriorate and increase simultaneously, then the bottleneck level for latency anomalies is determined to be located in the back-end NAND control layer, and the root cause of the latency anomaly is the performance degradation of the NAND media access link.

[0119] In this embodiment, when at least one preset reporting trigger condition is met, the delay statistics log of the corresponding time slice is recorded and reported to the host;

[0120] Specifically, the triggering, recording, and reporting of latency statistics logs are executed by the log reporting module within the non-volatile storage controller. A tiered triggering mechanism is used to proactively report abnormal events, eliminating the need for host polling and reducing interface interaction overhead while ensuring timely operation and maintenance monitoring. When at least one preset reporting trigger condition is met, the log reporting module automatically records the complete latency statistics log for the corresponding time slice and reports the log data to the host system via the NVMe standard asynchronous event reporting mechanism.

[0121] The preset reporting trigger conditions include the following three categories, and meeting any one of them will trigger log recording and reporting:

[0122] 1. Within any time slice, if the total latency of a single command exceeds a preset total command reporting threshold, a report is triggered. In this embodiment, the preset total command reporting threshold is 3 milliseconds; if the total latency of a host read command is 3.2 milliseconds, then this type of reporting condition is met.

[0123] 2. Within any time slice, if the proportion of total latency corresponding to commands with a high latency level exceeds a preset overall high latency proportion reporting threshold, a report is triggered. In this embodiment, the preset overall high latency proportion reporting threshold is 15%; within a single time slice of 100 milliseconds, if the proportion of total latency corresponding to commands with a high latency level is 17%, then this type of reporting condition is met.

[0124] 3. If, within a predetermined number of consecutive time slices, the overall total latency percentage corresponding to commands with a high latency level exceeds a predetermined continuous latency percentage reporting threshold, a report is triggered. In this embodiment, the predetermined number of consecutive time slices is 5, and the predetermined continuous latency percentage reporting threshold is 12%. If, within 5 consecutive time slices, the overall total latency percentage corresponding to commands with a high latency level is 13%, 14%, 13.5%, 15%, and 14.2% respectively, and the overall total latency percentage of all time slices exceeds the predetermined continuous latency percentage reporting threshold, then it is determined to be in a state of continuous latency degradation, and the reporting condition for this type is met.

[0125] The latency statistics log includes at least the percentage of total latency for each tenant identifier and command type, the rate of change of the percentage of total latency for each type, the percentage of latency for each sub-node, and the rate of change of latency for each sub-node within the corresponding time slice. In this embodiment, the logs are grouped and stored according to tenant identifier and command type. Each group of data fully covers the corresponding percentages and rates of change for low, medium, and high latency levels, providing fine-grained data support for the host side to locate abnormal tenants and analyze the causes of anomalies. In addition to core statistical data, the latency statistics log may also include the corresponding time slice number, the type of reporting trigger condition, and the index identifier of the corresponding non-volatile storage controller status snapshot, facilitating fault tracing and root cause analysis by the operations and maintenance side. The format of the latency statistics log conforms to the OCP storage telemetry specification requirements.

[0126] In summary, this invention, by synchronously collecting total command latency and child node latency, can accurately distinguish the latency bottleneck levels of the front-end scheduling layer and the back-end NAND control layer. This solves the problem of existing solutions only being able to count end-to-end total latency and unable to pinpoint the root cause of anomalies, providing accurate data for targeted performance optimization. By employing a time-slicing mechanism to count the proportion and rate of change of each latency level, combined with static threshold judgment and dynamic trend analysis, it overcomes the limitations of traditional static alarms, capturing continuous latency degradation trends and enabling early prediction and intervention of latency anomalies, thus solving the problem of alarm lag. It supports parallel statistics of multi-dimensional latency data by tenant identifier and command type, enabling precise identification of abnormal tenants and abnormal command types in scenarios with multiple tenants sharing storage resources. This achieves tenant-level fine-grained performance management, fully ensuring the service quality of core businesses. By combining non-volatile storage controller state snapshots to pinpoint specific root causes and matching differentiated adaptive adjustment strategies, the system replaces globally uniform coarse-grained scheduling, improving the accuracy of performance tuning. In conjunction with a tiered triggering latency statistics log reporting mechanism, it outputs standardized operation and maintenance data, balancing anomaly detection capabilities with monitoring overhead control, and adapting to enterprise-level storage operation and maintenance needs.

[0127] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any other combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product, which includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0128] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0129] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for delay monitoring and adaptive adjustment within a non-volatile memory controller, characterized in that, include: Receive standardized or custom configuration commands and configure monitoring parameters, including: monitoring trigger mode, tenant identifier, command type, duration and number of time slices, and delay interval division rules; The system monitors commands based on a monitoring trigger mode, calculates the total command latency and child node latency of each command, and determines the total latency level and child node latency level of each command based on latency interval division rules. Based on the duration and number of time slices, within each consecutive time slice, the system simultaneously counts the number of commands with each tenant identifier and command type, considering both the total latency level and child node latency level. Based on the number of commands, the system calculates the percentage of total command latency and the percentage of command child node latency. Finally, it compares the percentage of total command latency and the percentage of command child node latency between the current time slice and the previous adjacent time slice to obtain the rate of change of the percentage of total command latency and the rate of change of the percentage of command child node latency. Based on the percentage of total command latency, the rate of change of the percentage of total command latency, or the total command latency of a single command, an early warning signal is generated through preset latency judgment conditions; based on the early warning signal, a snapshot of the non-volatile memory controller status is generated. Based on the percentage of total command latency, the percentage of command sub-node latency, the rate of change of the percentage of total command latency, the rate of change of the percentage of command sub-node latency, and the state snapshot of the non-volatile memory controller, the root cause of latency anomalies is determined, and the non-volatile memory controller is adaptively adjusted.

2. The delay monitoring and adaptive adjustment method within a non-volatile memory controller according to claim 1, characterized in that, The monitoring triggering modes include: Based on the OCP specification, monitoring can be started within a preset trigger time period or at regular intervals according to a custom monitoring time period preset by the firmware.

3. The delay monitoring and adaptive adjustment method within a non-volatile memory controller according to claim 1, characterized in that, The calculation of the total command delay and child node delay includes: When a command begins execution, the corresponding tenant identifier and command type are parsed, a command tag is assigned to the command, and the total command start timestamp when the command arrives at the non-volatile storage controller interface is recorded. The tenant identifier, command type, and command tag are then combined and written into a preset command record table. When a command is sent to the NAND controller, the start timestamp of the command's child nodes is recorded, and based on the command tag, the start timestamp of the command's child nodes is written into the command record table. When a command is completed from the NAND controller, record the end timestamp of the command's child nodes; when the command is completed and the host is notified that the command is completed, record the total command end timestamp. Using the command tag of the command as an index, query the command record table to obtain the start timestamp and the start timestamp of the child nodes; subtract the total command start timestamp from the total command end timestamp to obtain the total command delay; subtract the child node start timestamp from the child node end timestamp of the command to obtain the child node delay of the command.

4. The delay monitoring and adaptive adjustment method within a non-volatile memory controller according to claim 3, characterized in that, The method for determining the total delay level and child node delay level of a command based on delay interval division rules includes at least the following: When the total delay of a command does not exceed the first preset total delay threshold, the total delay level of the command is confirmed as a low-latency command. When the total delay of a command exceeds the first preset total delay threshold but does not exceed the second preset total delay threshold, the total delay level of the command is confirmed as a medium delay command. When the total delay of a command exceeds the second preset total delay threshold, the total delay level of the command is confirmed as a high-delay command. When the delay of a child node does not exceed the first preset child node delay threshold, the delay level of the child node of the command is confirmed as a low-delay command. When the delay of a child node exceeds the first preset child node delay threshold but does not exceed the second preset child node delay threshold, the delay level of the child node of the command is confirmed to be a medium delay command. When the delay of a child node exceeds the second preset child node delay threshold, the delay level of the child node of the command is confirmed as a high delay command.

5. The delay monitoring and adaptive adjustment method within a non-volatile memory controller according to claim 4, characterized in that, The number of commands for each tenant identifier and under any command type, including the total latency level and the child node latency level, are counted in parallel. And based on the number of commands, calculate the total command latency percentage and the command sub-node latency percentage, including: The total command latency percentage includes the overall total latency percentage under different total latency levels and the total latency percentage of a single category; The overall total latency percentage is the ratio of the total number of commands of all tenant identifiers and all command types corresponding to each total latency level within a single time slice to the total number of commands within that time slice. The single-category total latency percentage is the ratio of the number of commands of any tenant identifier and any command type corresponding to each total latency level within a single time slice to the total number of commands of that tenant identifier and any command type within that time slice. The command sub-node latency percentage includes the overall sub-node latency percentage under different sub-node latency levels and the latency percentage of a single type of sub-node; The overall sub-node latency ratio is the ratio of the total number of commands of all tenant identifiers and all command types corresponding to the latency level of each sub-node within a single time slice to the total number of commands within that time slice. The single-type sub-node latency ratio is the ratio of the number of commands of any tenant identifier and any command type corresponding to the latency level of each sub-node within a single time slice to the total number of commands of that tenant identifier and any command type within that time slice.

6. The delay monitoring and adaptive adjustment method within a non-volatile memory controller according to claim 5, characterized in that, The obtained percentage change rate of total command latency and percentage change rate of command sub-node latency include: The percentage change in total command latency includes the percentage change in overall total latency and the percentage change in total latency for a single category; The difference between the total total latency percentage of each total latency level in the current time slice and the corresponding total total latency percentage in the previous adjacent time slice is divided by the duration of the current time slice to obtain the corresponding rate of change of the total total latency percentage. The difference between the percentage of total latency for each tenant identifier and command type under each total latency level within the current time slice and the percentage of total latency for the corresponding single category in the previous adjacent time slice is divided by the duration of the current time slice to obtain the corresponding percentage change rate of total latency for the single category. The change rate of the delay ratio of the command sub-nodes includes the change rate of the overall delay ratio of sub-nodes and the change rate of the delay ratio of a single type of sub-node; The difference between the overall child node latency percentage under each child node latency level in the current time slice and the corresponding overall child node latency percentage in the previous adjacent time slice is divided by the duration of the current time slice to obtain the corresponding rate of change of the overall child node latency percentage. The difference between the single-type child node latency percentage under each child node latency level for any tenant identifier and any command type within the current time slice and the corresponding single-type child node latency percentage in the previous adjacent time slice is divided by the duration of the current time slice to obtain the corresponding single-type child node latency percentage change rate.

7. The delay monitoring and adaptive adjustment method within a non-volatile memory controller according to claim 6, characterized in that, The preset delay determination conditions include at least the following: If, within a single time slice, the total delay of any command exceeds the preset single command delay warning threshold, or if, within a single time slice, the proportion of the total delay corresponding to the preset total delay level exceeds the preset total proportion warning threshold, or if, within a consecutive preset number of time slices, the rate of change of the proportion of the total delay corresponding to commands with a high delay level exceeds the preset deterioration rate warning threshold, then a warning signal is generated. Based on the warning signal, a snapshot of the non-volatile memory controller's state is generated according to the internal state information of the non-volatile memory controller. The state information includes, but is not limited to, the running state of the state machine inside the non-volatile memory controller, the request queue depth, the completion queue depth, error status information, the number of error correction decoding iterations, and the background task type and execution status.

8. The delay monitoring and adaptive adjustment method within a non-volatile memory controller according to claim 7, characterized in that, The process of determining the root cause of delay anomalies and adaptively adjusting the non-volatile memory controller includes: a. When, within a consecutive preset number of time slices, the overall total latency percentage of commands with a high latency level exceeds the preset overall total latency percentage abnormal threshold or the corresponding overall total latency percentage change rate exceeds the preset overall total deterioration rate threshold, and the overall child node latency percentage of commands with a high latency level does not exceed the preset overall child node latency percentage abnormal threshold or the corresponding overall child node latency percentage change rate does not exceed the preset overall child node deterioration rate threshold, then the latency abnormality bottleneck level is confirmed to be at the front-end scheduling layer. The abnormal tenant is determined by combining the single-type total latency percentage of different tenants, the actual load and the allocated bandwidth percentage threshold, and the command scheduling weight or scheduling priority of the abnormal tenant is adjusted to reduce the bandwidth quota. b. If, within a consecutive preset number of time slices, the overall total latency percentage of commands with a high latency level exceeds a preset overall total latency percentage abnormal threshold or the corresponding overall total latency percentage change rate exceeds a preset overall total degradation rate threshold, and the overall child node latency percentage of corresponding child node commands with a high latency level exceeds a preset overall child latency percentage abnormal threshold and the corresponding overall child node latency percentage change rate exceeds a preset overall child degradation rate threshold, then the latency abnormality bottleneck level is determined to be at the backend NAND control layer. If the non-volatile storage controller status snapshot shows that the background is currently performing garbage collection or cold data migration tasks, then the background task priority is adjusted to prioritize responding to foreground IO requests. If the non-volatile storage controller status snapshot shows that the number of error correction decoding iterations of data read from NAND exceeds a preset abnormal threshold, then the data in the corresponding physical storage block is moved to an available physical storage block with an erase count lower than a preset erase count threshold and a bit error rate within a preset normal range.

9. The delay monitoring and adaptive adjustment method within a non-volatile memory controller according to claim 8, characterized in that, The total command latency percentage also includes the total interval distribution percentage, and the change rate of the total command latency percentage also includes the change rate of the total interval distribution percentage; the total interval distribution percentage is the set of the overall total latency percentages corresponding to all total latency levels arranged in the order of latency intervals, and the change rate of the total interval distribution percentage is the set of the change rates of the overall total latency percentages corresponding to each total latency level between adjacent time slices; the total interval distribution percentage and the change rate of the total interval distribution percentage are used to generate latency distribution histograms, conduct latency distribution trend analysis, and verify the adaptive adjustment effect; The command sub-node latency percentage also includes the sub-interval distribution percentage, and the command sub-node latency percentage change rate also includes the sub-interval distribution percentage change rate. The sub-interval distribution percentage is a set of the overall sub-node latency percentages corresponding to all sub-node latency levels arranged in the order of latency intervals, and the sub-interval distribution percentage change rate is a set of the overall sub-node latency percentage change rates corresponding to each sub-node latency level between adjacent time slices. The sub-interval distribution percentage and the sub-interval distribution percentage change rate are used to generate a NAND path latency distribution histogram, conduct NAND path latency distribution trend analysis, verify the adaptive adjustment effect of the NAND layer, and assist in locating the bottleneck level of latency anomalies.

10. The delay monitoring and adaptive adjustment method within a non-volatile memory controller according to claim 7, characterized in that, When at least one preset reporting trigger condition is met, the delay statistics log for the corresponding time slice is recorded and reported to the host; the preset reporting trigger conditions include: Within any time slice, the total delay of a single command exceeds the preset total command reporting threshold; Within any time slice, the overall total latency percentage of commands with a total latency level of high latency exceeds the preset overall high latency percentage reporting threshold; Within a consecutive preset number of time slices, the overall total latency percentage of commands with a high latency level exceeds the preset consecutive latency percentage reporting threshold. The latency statistics log includes at least the percentage of total latency for each tenant identifier and command type, the rate of change of the percentage of total latency for each type, the percentage of latency for each sub-node, and the rate of change of the percentage of latency for each sub-node within the corresponding time slice.

Images