Storage performance bottleneck cross-layer intelligent positioning method and system of full flash memory storage system

Abstract

The invention relates to the technical field of performance optimization of all-flash memory storage systems, and particularly discloses a storage performance bottleneck cross-layer intelligent positioning method and system of an all-flash memory storage system.The method comprises the steps that cross-layer performance index data of an all-flash memory distributed storage system is collected; establishing a hierarchical structure and a call chain; performing index association analysis based on the cross-layer performance index data to obtain an association result between the performance indexes of each layer; constructing a cross-layer performance association view based on the hierarchical structure, the call chain and the association result between the performance indexes of each layer; modeling based on the cross-layer performance association view to obtain a cross-layer performance association model; and when an exception occurs, identifying a performance bottleneck node under the current exception based on the cross-layer performance correlation model. According to the method, the internal state of the SSD is monitored, cross-layer tracking is combined, and performance bottleneck diagnosis is carried out for a full-flash-memory distributed storage system.

Description

Technical Field

[0001] The present invention relates to the technical field of all-flash storage system performance optimization, and in particular to a method and system for intelligently locating storage performance bottlenecks across layers of an all-flash storage system. Background Art

[0002] Traditional storage performance monitoring methods, primarily designed for HDDs, fail to adequately address issues unique to all-flash storage, such as garbage collection (GC) performance degradation and write amplification. They are unable to detect in real time the impact of GC on I / O latency. Random write performance and the internal data layout of solid-state drives (SSDs) significantly impact overall performance in all-flash systems. However, existing methods lack in-depth monitoring of SSD internal states, making it difficult to identify bottlenecks caused by data fragmentation or metadata management.

[0003] In addition, the bottlenecks of all-flash distributed storage may be distributed at multiple levels, such as the protocol stack and network, making it difficult for traditional tools to conduct cross-layer correlation analysis; the performance of all-flash systems is highly dependent on the load type, and the performance of the same all-flash array varies significantly in different application scenarios, but existing tools fail to effectively distinguish workload characteristics. Summary of the Invention

[0004] In response to the above problems, the first purpose of the present invention is to provide a cross-layer intelligent location method for storage performance bottlenecks in an all-flash storage system. The method monitors the internal status of the SSD and combines cross-layer tracking to diagnose performance bottlenecks in the all-flash distributed storage system. At the same time, it dynamically distinguishes load types and automatically adjusts data storage strategies to enhance workload perception capabilities. It achieves end-to-end low latency by eliminating protocol and network transmission bottlenecks. It introduces AI-driven automated tuning to predict future storage performance bottleneck nodes based on real-time data, and optimizes data layout, caching strategies, and resource allocation.

[0005] A second object of the present invention is to provide a cross-layer intelligent location system for storage performance bottlenecks of an all-flash storage system.

[0006] The first technical solution adopted by the present invention is: a method for intelligently locating storage performance bottlenecks across layers of an all-flash storage system, comprising the following steps:

[0007] S100: Real-time monitoring and collection of cross-layer performance indicator data of all-flash distributed storage systems;

[0008] S200: Establishing a hierarchical structure and call chain of an all-flash distributed storage system; performing indicator correlation analysis based on the cross-layer performance indicator data to obtain correlation results between performance indicators of each layer; and constructing a cross-layer performance correlation view based on the hierarchical structure and call chain and the correlation results between performance indicators of each layer.

[0009] S300: Modeling is performed based on the cross-layer performance correlation view to obtain a cross-layer performance correlation model; a conditional probability table is set for each indicator node in the cross-layer performance correlation model, and when an abnormality occurs in the performance indicator of a certain layer, the posterior probability of each indicator node under the abnormal condition is calculated based on the conditional probability table of each indicator node; the posterior probabilities of all indicator nodes in the cross-layer performance correlation model are traversed, and the indicator node with the largest posterior probability is the performance bottleneck node under the current abnormality.

[0010] Preferably, the cross-layer performance indicator data in step S100 includes performance indicator data of the storage layer, network layer, protocol layer and application layer.

[0011] Preferably, the step S200 includes:

[0012] Collect FTL metadata, perform I / O path tracing based on the FTL metadata, and record I / O path tracing data; establish a hierarchical structure and call chain of an all-flash distributed storage system based on the I / O path tracing data; wherein the I / O path tracing data includes one or more of the following data: an operation name, timestamp, duration, and general description information of each I / O request.

[0013] Preferably, the step S200 includes:

[0014] The timestamp alignment technology is used to timestamp-align the cross-layer performance indicator data. When an abnormality occurs in the performance indicator data of a certain layer, the performance indicator data of other layers at the same timestamp are analyzed to see whether an abnormality occurs. If an abnormality occurs in the performance indicator data of other layers at the same timestamp, the abnormal performance indicator data of the layer is correlated with the abnormal performance indicator data of other layers to obtain the correlation results between the performance indicators of each layer.

[0015] Preferably, the step S300 includes:

[0016] The indicators of each layer in the cross-layer performance correlation view are used as indicator nodes, and the correlation relationship between the indicators is used as the dependency relationship. Based on the Bayesian network, the indicator nodes are modeled through directed edge connections to obtain a cross-layer performance correlation model.

[0017] Preferably, the conditional probability table in step S300 is the probability distribution of each indicator node under different parent node states;

[0018] The posterior probability of each indicator node under abnormal conditions is calculated using the following formula:

[0019]

[0020] Where, P(X i |A=Abnormal) is the performance indicator of a certain layer. When an abnormality occurs, X i The posterior probability of P(A = abnormal|X i ) is X i The probability that a certain layer's performance indicator A will be abnormal when an abnormality occurs; P(X i ) is X i The probability of an anomaly; P(A=anomaly) is the probability of an anomaly occurring in a performance indicator A of a certain layer.

[0021] Preferably, the step S300 further includes:

[0022] Based on the cross-layer performance indicator data, the current load type is identified, and based on the performance bottleneck node under the current abnormality, the layer where the performance bottleneck node is located is obtained; and performance optimization is performed based on the layer where the performance bottleneck node is located and the current load type.

[0023] Preferably, the cross-layer intelligent positioning method for all-flash storage performance bottlenecks also includes step S400: collecting traffic data of the all-flash distributed storage system and external events that affect the load; predicting future storage performance bottleneck nodes based on the traffic data, external events that affect the load and the cross-layer performance indicator data; and optimizing the performance of the all-flash distributed storage system in advance based on the future storage performance bottleneck nodes.

[0024] The second technical solution adopted by the present invention is: a cross-layer intelligent location system for storage performance bottlenecks in an all-flash storage system, comprising a data collection and cross-layer monitoring module and an analysis and decision module;

[0025] The data collection and cross-layer monitoring module is used to: monitor and collect cross-layer performance indicator data of the all-flash distributed storage system in real time, perform indicator correlation analysis based on the cross-layer performance indicator data, obtain correlation results between performance indicators of each layer; and establish a hierarchical structure and call chain; and construct a cross-layer performance correlation view based on the hierarchical structure and call chain and the correlation results between performance indicators of each layer;

[0026] The analysis and decision module is used to: perform modeling based on the cross-layer performance correlation view to obtain a cross-layer performance correlation model; set a conditional probability table for each indicator node in the cross-layer performance correlation model, and when an abnormality occurs in the performance indicator of a certain layer, calculate the posterior probability of each indicator node under the abnormal condition based on the conditional probability table of each indicator node; traverse the posterior probabilities of all indicator nodes in the cross-layer performance correlation model, and the indicator node with the largest posterior probability is the performance bottleneck node under the current abnormality; and is also used to identify the current load type based on the cross-layer performance indicator data, and obtain the level of the performance bottleneck node based on the performance bottleneck node under the current abnormality.

[0027] Preferably, the all-flash storage performance bottleneck cross-layer intelligent positioning system further includes: a dynamic strategy execution and optimization module and an AI-driven closed-loop tuning module;

[0028] The dynamic strategy execution and optimization module is used to: perform performance optimization based on the level of the performance bottleneck node and the current load type;

[0029] The AI-driven closed-loop tuning module is used to: collect traffic data of the all-flash distributed storage system and external events that affect the load; predict future storage performance bottleneck nodes based on the traffic data, external events that affect the load and the cross-layer performance indicator data; and optimize the performance of the all-flash distributed storage system in advance based on the future storage performance bottleneck nodes.

[0030] Beneficial effects of the above technical solution:

[0031] (1) The cross-layer intelligent positioning method for all-flash storage performance bottlenecks based on differentiated load perception disclosed in the present invention covers full-stack monitoring from the storage layer, network layer, protocol layer to the application layer. On the basis of distinguishing workload characteristics, it conducts cross-layer correlation analysis of the performance bottlenecks of the all-flash distributed storage system. Through AI-driven automatic tuning, it predicts bottlenecks based on real-time data, optimizes data layout, cache strategy and resource allocation, breaks the limitations of traditional single-layer optimization, achieves global performance optimization, and solves the performance bottleneck problem caused by load dynamics and software stack complexity in distributed storage systems.

[0032] (2) The cross-layer intelligent location method for storage performance bottlenecks in all-flash storage systems based on differentiated load perception proposed in the present invention monitors the internal status of SSDs, including indicators such as wear, GC frequency, and write amplification factor, and combines cross-layer tracking to diagnose performance bottlenecks in all-flash distributed storage systems; at the same time, it dynamically distinguishes load types and automatically adjusts data storage strategies, thereby enhancing workload perception capabilities; achieves end-to-end low latency by eliminating protocol and network transmission bottlenecks; introduces AI-driven automated tuning, predicts future storage performance bottleneck nodes based on real-time data, and optimizes data layout, caching strategies, and resource allocation.

[0033] (3) The cross-layer intelligent positioning system for storage performance bottlenecks of all-flash storage systems based on differentiated load perception disclosed in the present invention realizes cross-layer monitoring of "bottom layer-protocol-network" through data collection and cross-layer monitoring modules, analysis and decision modules, dynamic strategy execution and optimization modules, and AI-driven closed-loop tuning modules. It perceives differentiated load types and dynamically adapts based on them, innovates at the protocol layer to achieve low-latency transmission, and realizes predictive tuning through AI closed loop, breaking through the performance bottleneck positioning of traditional storage systems.

[0034] (4) The present invention avoids unpredictable delays caused by resource contention under mixed workloads by dynamically adjusting the priority scheduling mechanism. In addition, by introducing a machine learning model to predict bottlenecks and automatically adjust parameters, the situation where the performance of the all-flash system is more affected by real-time load fluctuations is effectively addressed. At the same time, the present invention combines more fine-grained indicators to achieve end-to-end tracking.

[0035] (5) The performance of an all-flash system is highly dependent on the load type. The performance of the same all-flash array varies significantly in different application scenarios. However, existing tools fail to effectively distinguish workload characteristics and cannot actually guide the improvement of all-flash storage performance. Compared with the existing technology, the present invention supports dynamic differentiation of load types and automatically adjusts data paths, thereby enhancing workload perception capabilities. The optimization strategy formed based on this can effectively improve the performance of storage systems based on all-flash arrays. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a process for intelligently locating storage performance bottlenecks across layers of an all-flash storage system according to an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the structure of a cross-layer intelligent location system for storage performance bottlenecks in an all-flash storage system provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following detailed description of the embodiments of the present invention is provided in conjunction with the accompanying drawings and examples. The following detailed description of the embodiments and the accompanying drawings are intended to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention. That is, the present invention is not limited to the preferred embodiments described, and the scope of the present invention is defined by the claims.

[0039] In the description of the present invention, it should be noted that, unless otherwise specified, “plurality” means two or more; the terms “first”, “second”, etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance; for ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0040] Example 1

[0041] like Figure 1 As shown, an embodiment of the present invention provides a method for intelligently locating storage performance bottlenecks across layers of an all-flash storage system, comprising the following steps:

[0042] S100: Real-time monitoring and collection of cross-layer performance indicator data of all-flash distributed storage systems;

[0043] Real-time monitoring and collection of cross-layer performance indicator data of the all-flash distributed storage system, including performance indicator data of the storage layer, network layer, protocol layer and application layer;

[0044] Collecting storage layer performance indicator data includes: real-time collection of solid-state drive (SSD) internal status data through the SMART interface or custom drivers. The SSD internal status data includes but is not limited to: wear, garbage collection (GC) frequency, write amplification factor, P / E cycles (erase cycles), SSD remaining service life, and spatial locality;

[0045] Collecting network layer performance indicator data includes: collecting data through command line tools (such as Ping, Iperf), and the network layer indicators include but are not limited to: packet loss rate, bandwidth utilization, latency, etc.;

[0046] Collecting protocol layer performance indicator data includes: collecting protocol layer performance indicator data through the PCIe bus and the built-in logging function of the NVMe SSD, the protocol layer indicators include but are not limited to NVMe command queue depth, command latency, number of retries, etc.;

[0047] Collecting application-layer performance metrics includes: ① Tracking the data block and packet sizes of each I / O operation through call parameters and analyzing their distribution patterns; ② Capturing I / O operation types through tracing tools (such as Recorder and Jaeger); ③ Using Fourier transforms to detect the dominant frequency of time series data, thereby detecting the periodic characteristics of I / O requests (such as the fixed interval between batch writes);

[0048] The application layer performance indicator data includes but is not limited to: data block size, data packet size, I / O operation type, periodic characteristics of I / O requests, read and write latency, read and write ratio, throughput, IOPS, response time, etc.

[0049] S200: Establishing a hierarchical structure and call chain of an all-flash distributed storage system; performing indicator correlation analysis based on the cross-layer performance indicator data to obtain correlation results between performance indicators of each layer; and constructing a cross-layer performance correlation view based on the hierarchical structure and call chain and the correlation results between performance indicators of each layer.

[0050] (1) Establishing the hierarchical structure and call chain of the all-flash distributed storage system includes: collecting FTL (flash translation layer) metadata, performing I / O path tracing based on the FTL metadata, and recording I / O path tracing data; establishing the hierarchical structure and call chain of the all-flash distributed storage system based on the I / O path tracing data.

[0051] An embedded probe is integrated into the SSD controller to collect FTL (Flash Translation Layer) metadata in real time. For example, the embedded probe is a probe that combines bitmap technology and FTL algorithm verification functions. Probe code is inserted into key FTL functions (such as ftl_map_update() and gc_trigger()) to record FTL (Flash Translation Layer) metadata changes. The FTL metadata includes, for example, P / E cycles and block erase counts.

[0052] Based on the FTL metadata and in conjunction with a distributed tracing framework (e.g., OpenTelemetry), an all-flash distributed storage system is implemented with end-to-end I / O path tracing from client requests to physical writes to the SSD, and I / O path tracing data is recorded. The I / O path tracing data includes, but is not limited to, the operation name, timestamp, duration, and general description information (i.e., general description information used to track the entire lifecycle of a request, including physical metadata, storage metadata, and computing metadata, and does not involve specific operations within the SSD) for each I / O request.

[0053] A hierarchical structure and call chain of an all-flash distributed storage system are established based on the I / O path tracing data; the hierarchical structure of the all-flash distributed storage system includes a storage layer, a network layer, a protocol layer, and an application layer.

[0054] (2) Normalizing the cross-layer performance indicator data into a time series and performing indicator correlation analysis to obtain correlation results between performance indicators at each layer, including:

[0055] The timestamp alignment technology is used to timestamp-align the cross-layer performance indicator data. When an abnormality occurs in the performance indicator data of a certain layer, the performance indicator data of other layers at the same timestamp are analyzed to see whether an abnormality occurs. If an abnormality occurs in the performance indicator data of other layers at the same timestamp, the abnormal performance indicator data of the layer is correlated with the abnormal performance indicator data of other layers to obtain the correlation results between the performance indicators of each layer.

[0056] For example, the storage layer performance indicator data (i.e., SSD internal status data) can be associated with the network layer performance indicator data (packet loss rate, bandwidth utilization) and the protocol layer performance indicator data (NVMe command queue depth, command latency) to obtain the correlation results between the performance indicators of each layer.

[0057] S300: Modeling is performed based on the cross-layer performance correlation view to obtain a cross-layer performance correlation model; a conditional probability table is set for each indicator node in the cross-layer performance correlation model; when an abnormality occurs in a performance indicator of a certain layer, the posterior probability of each indicator node under the abnormal condition is calculated based on the conditional probability table of each indicator node; the posterior probabilities of all indicator nodes in the cross-layer performance correlation model are traversed, and the indicator node with the largest posterior probability is the performance bottleneck node under the current abnormality;

[0058] Based on the cross-layer performance correlation view, the dependency relationships between indicators at different layers are modeled using a Bayesian network to identify bottleneck sources. This includes the following sub-steps:

[0059] S310: Using the indicators of each layer in the cross-layer performance correlation view as indicator nodes and the correlation relationships between the indicators as dependency relationships, modeling the indicator nodes by connecting them with directed edges based on a Bayesian network, thereby obtaining a cross-layer performance correlation model;

[0060] Considering the indicators of each layer as indicator nodes, the dependencies (i.e., the correlation between indicators in the cross-layer performance correlation view) are connected through directed edges. The joint probability distribution can be decomposed into the following formula:

[0061]

[0062] Where, P(X1,X2,…,X n ) is the joint probability distribution of all indicator nodes; X i is the index node, which is a random variable; Parents(X i ) is the index node X i The parent node set of represents the upstream layer indicators that directly affect the indicator. For example, the response time of the application layer may depend on the number of retries at the protocol layer and the delay at the network layer. n is the total number of indicator nodes, which can determine the dimension of the joint probability, that is, the number of conditional probabilities that need to be multiplied. i is the position of the currently processed indicator node in the sequence, which is used to decompose the joint probability.

[0063] S320: Set a conditional probability table for each indicator node in the cross-layer performance correlation model. When an abnormality occurs in the performance indicator of a certain layer, calculate the posterior probability of each indicator node under the abnormal condition based on the conditional probability table of each indicator node; traverse the posterior probabilities of all indicator nodes in the cross-layer performance correlation model, and the indicator node with the largest posterior probability is the performance bottleneck node under the current abnormality.

[0064] Each index node X i, define its conditional probability table to quantify the intensity of the parent node's influence on it; for example, for a certain layer's performance indicator A (such as application layer response time), it may depend on both the number of retries (R) at the protocol layer and the delay (N) at the network layer. The probability distribution (i.e., conditional probability table) of each indicator node (such as application layer response time) under different parent node states (number of retries at the protocol layer and delay at the network layer) is defined as follows:

[0065]

[0066] Where P(A|N,R) is the probability distribution of the performance indicator node A at a certain layer under the influence of the number of retries at the protocol layer and the delay at the network layer.

[0067] When an abnormality is detected in the performance indicators of a certain layer, for example, when an abnormality is detected in the application layer response time (when the response time exceeds both the absolute threshold and the relative threshold of the average response time, it is considered abnormal), the posterior probability of each indicator node under the abnormal condition (abnormal application layer response time) is calculated. The posterior probability of each indicator node under the abnormal condition is calculated using the following formula:

[0068]

[0069] Where, P(X i |A=Abnormal) is the performance indicator of a certain layer. When an abnormality occurs, X i The posterior probability of P(A = abnormal|X i ) is X i The probability that a certain layer's performance indicator A will be abnormal when an abnormality occurs; P(X i ) is X i The probability of an anomaly; P(A=anomaly) is the probability of an anomaly occurring in a performance indicator A of a certain layer; A is, for example, the application layer response time.

[0070] The posterior probabilities of nodes on all paths in the cross-layer performance correlation model are traversed, and the node with the largest posterior probability is used as the performance bottleneck node under the current anomaly, thereby identifying and locating the performance bottleneck node under the current anomaly.

[0071] Furthermore, in one embodiment, step S300 further includes: identifying a current load type based on the cross-layer performance indicator data, and obtaining a layer at which the performance bottleneck node is located based on the performance bottleneck node under the current abnormality; and performing performance optimization based on the layer at which the performance bottleneck node is located and the current load type.

[0072] Performance optimization is performed based on the level of the performance bottleneck node and the current load type, including:

[0073] Identify the current load type based on the cross-layer performance indicator data; build a load fingerprint library, match the strategy with the load fingerprint library based on the layer where the performance bottleneck node is located and the current load type to obtain a performance optimization strategy; dynamically optimize and adjust the current storage strategy based on the performance optimization strategy.

[0074] The identifying of the current load type based on the cross-layer performance indicator data includes:

[0075] Based on the I / O data block / packet size distribution, read-write ratio, spatial locality and other indicators and load types in the cross-layer performance indicator data, the current load type is identified in real time through an online load classification algorithm (for example, using K-Means, Gaussian mixture model (GMM) and other clustering algorithms to classify workloads); the load types include small block random write, small block random read, large block sequential write, large block sequential read, mixed read and write, multi-threaded / concurrent load, OLTP (transaction processing), OLAP (analytical processing), read-write intensive, compute intensive, deduplication sensitive, bandwidth sensitive, frequent updates, delay sensitive and other types.

[0076] Building a load fingerprint library involves discretizing or mapping the load type into a discrete set of states, designing performance optimization strategies at different levels based on the specific load type, designing reward functions centered around optimizing key performance indicators under different load types, optimizing performance strategies based on Q-learning, and storing the load type, performance optimization strategy, and its corresponding Q value as a load fingerprint to form a load fingerprint library.

[0077] The load type is discretized or mapped into a discrete set of states (i.e., a state space). Based on the specific load type, performance optimization strategies are designed at different layers (storage layer, protocol layer, network layer, and application layer) (i.e., the action space is designed) to construct a state-action value function table, which is used to store the expected value of the cumulative reward (Q value) obtained by executing a specific action in a given state. The reward function design must be closely integrated with the key performance indicators of different load types. For read-write intensive loads, for example, the key performance indicators are throughput and latency, so the reward function can be designed as a weighted combination of response time and resource consumption. After defining the state space and action space, the state-action value function table is initialized. At each time step, the operation of "selecting an action based on the current state - executing the action and observing the new state and immediate reward - updating the Q value" is repeated until convergence. These load states (i.e., load types), performance optimization strategies, and their corresponding Q values are stored as load fingerprints, forming a load fingerprint library. This allows for rapid matching of load types with performance optimization strategies at the layer where performance bottleneck nodes reside. Among them, the design of performance optimization strategies includes: scheduling priority adjustment, prefetch strategy selection (prefetch strategy is a method of loading data into the cache in advance before the data is accessed to reduce waiting time and improve access efficiency), turning off compression, disabling compression, turning off deduplication, and zero-copy transmission.

[0078] Based on the current load type and the level of the performance bottleneck node output by the load classifier, the load fingerprint library is combined to perform policy matching to obtain a performance optimization strategy, and the current storage strategy is dynamically optimized and adjusted based on the performance optimization strategy to dynamically optimize the performance of the all-flash distributed storage system;

[0079] For example, if the current load type is deduplication-sensitive and the performance bottleneck node is located at the storage layer, the optimal data path strategy is to disable compression to reduce latency and thus optimize the storage layer.

[0080] If the current load type is bandwidth-sensitive and the performance bottleneck node is located at the network layer or protocol layer, the optimal data path strategy is to enable zero-copy transmission, thereby optimizing the network layer and protocol layer.

[0081] If the current load type is latency-sensitive and the performance bottleneck node is located at the storage layer or network layer, the optimal data path strategy is to disable compression to reduce data transmission time and latency, thereby optimizing the storage layer or network layer.

[0082] If the current load type is small block random writes and frequent updates, and the performance bottleneck node is located at the storage layer, the optimal data path strategy is to disable deduplication to reduce SSD write amplification and thus optimize the storage layer.

[0083] Furthermore, in one embodiment, step S300 further includes: if the performance indicator corresponding to the performance bottleneck node is at a protocol layer, merging multiple I / O requests through a single all-flash distributed storage system call to merge multiple buffers into one atomic operation;

[0084] Atomic operation support is integrated into the protocol stack of the protocol layer, supporting atomic scatter-gather operations. Multiple small I / O requests are merged through a single system call, and multiple buffers are merged into one atomic operation, reducing the repeated encapsulation of protocol headers and thus reducing end-to-end latency.

[0085] Furthermore, in one embodiment, step S300 also includes: isolating resources based on the current load type and task priority, reserving independent QoS channels for critical loads, and avoiding low-priority tasks from preempting resources; and real-time monitoring of the load (such as IOPS, latency) of the storage nodes of the all-flash distributed storage system. If the load of any storage node exceeds the threshold, the hot data of the storage node is automatically migrated to the idle node; and elastically increasing storage nodes according to business growth, thereby expanding the storage pool.

[0086] Furthermore, in one embodiment, step S400 is also included: collecting traffic data of the all-flash distributed storage system and external events that affect the load; predicting future storage performance bottleneck nodes based on the traffic data, external events that affect the load and cross-layer performance indicator data; and optimizing the performance of the all-flash distributed storage system in advance based on the future storage performance bottleneck nodes.

[0087] (1) predicting a future storage performance bottleneck node based on the traffic data, external events affecting the load, and cross-layer performance indicator data, including: predicting the remaining service life of the SSD based on the cross-layer performance indicator data to obtain a predicted remaining service life (RUL) of the SSD; and predicting a future storage performance bottleneck node based on the predicted remaining service life of the SSD;

[0088] And based on the traffic data and external events that affect the load, the future load peak is predicted to obtain the load (such as IOPS, latency) predicted peak; if the load prediction peak is greater than or equal to the preset threshold, and the SSD predicted remaining service life is less than or equal to the set threshold, then the indicator corresponding to the load prediction peak and the SSD remaining service life are the future storage performance bottleneck node.

[0089] Among them, predicting the remaining service life of the SSD based on the cross-layer performance indicator data includes: inputting the storage layer performance indicator data in the cross-layer performance indicator data into the LSTM model, and monitoring the second-order derivative extreme point of the RUL prediction curve based on the LSTM model, thereby obtaining the SSD predicted life inflection point and the SSD predicted remaining service life (RUL).

[0090] Based on the cross-layer performance indicator data, the health prediction model converts the storage layer performance indicator data into a tensor covering the number of samples (i.e., data segments within a time window, such as the number of the past 120 hours), the time step (i.e., the sequence length of a single sample, such as 120 hours of data = 120 time steps, the time step is determined by the size of the sliding time window), and the number of features (the monitoring indicator dimension of each time step) according to the time window, and obtains a time series sample (number of samples × time step × number of features).

[0091] The time series samples are input into the LSTM model, and the LSTM model is used to output the RUL prediction curve; then the discrete difference method is used to calculate the second-order derivative of the RUL prediction curve (y(t+1)-2y(t)+y(t-1)) (where y(t) represents the SSD remaining service life predicted at the t-th time step, and t represents the time series index) to locate the extreme point. When zero crossing occurs or exceeds the adaptive threshold (such as ±3σ), it is determined to be an inflection point, thereby obtaining the SSD predicted life inflection point and the SSD predicted remaining service life (RUL); where zero crossing refers to the sign change of the second-order derivative of the RUL prediction curve from positive to negative or negative to positive, that is, crossing the zero point, and the corresponding curvature direction changes; σ is the standard deviation of the second-order derivative, and the dynamic threshold is set using the normal distribution 3σ principle. Exceeding ±3σ is considered an abnormal inflection point.

[0092] The predicted remaining SSD lifespan is correlated with performance metrics at each layer. Performance degradation factors, such as latency growth or IOPS decrease, are quantified based on the predicted SSD lifespan. If these performance degradation factors are greater than or equal to a set threshold, the SSD's predicted remaining lifespan represents a future storage performance bottleneck. For example, for every 10% of the SSD's predicted remaining lifespan consumed, there will be an 8-15% decrease in write IOPS. If the decrease in write IOPS is greater than or equal to the set threshold, the SSD's predicted remaining lifespan represents a future storage performance bottleneck.

[0093] Predicting future load peaks based on the traffic data and external events affecting the load, including:

[0094] 1) Obtain time-stamped traffic trend indicators and external events that affect load;

[0095] The collected traffic data of the all-flash distributed storage system is stored in time series and normalized to obtain normalized traffic data; and a traffic trend indicator with a timestamp is obtained based on the normalized traffic data, and the traffic trend indicator quantifies the degree to which the traffic deviates from the normal state; the traffic trend indicator = [current traffic value - normal baseline value) / normal baseline value] × 100%, and the normal baseline value is set according to different scenarios and can also be generated by analyzing historical traffic data.

[0096] External events that affect load include but are not limited to: business peak periods, system maintenance, etc.

[0097] 2) Encoding the external events that affect the load (such as business peaks, system maintenance, etc.) as features; and generating time series samples from the external events that affect the load through a sliding window;

[0098] The categorical events (such as fault alarms, etc.) in the external events are encoded using one-hot encoding, and the continuous events (such as the number of concurrent users, etc.) in the external events are directly input as features; the external events that affect the load are generated into time series samples through a sliding window.

[0099] 3) Aligning the timestamp-carrying traffic trend indicator (i.e., historical load) with the time series sample according to the timestamp to form a multidimensional input vector; inputting the multidimensional input vector into the LSTM to obtain the load prediction peak value.

[0100] Calculate the importance score (i.e., weight) α of each time step in the time series sample based on LSTM t =softmax(W q ·h t ), where α t Reflects the importance of the tth moment to the current prediction, h t is the hidden state vector (dimension d) at time t (i.e., a multidimensional input vector), which refers to the dynamic memory carrier maintained when processing time series data, carrying historical patterns and time-dependent information; W q is a learnable d×d parameter matrix used to extract key features in the hidden state;

[0101] Focusing on key moments based on the importance scores (weights) of each time step, weighting the hidden state vector (i.e., the multidimensional input vector) is then concatenated with the current multidimensional input vector to form a joint feature, thereby fully utilizing historical information and the current state;

[0102] The joint features are sent to the GLU for processing. First, information filtering is implemented through the sigmoid function to control the flow of each feature. At the same time, nonlinear transformation is performed to enhance the expressive power of the model. Finally, adaptive feature fusion is achieved through element-by-element multiplication. The final prediction value is output through the fully connected layer to obtain the load (such as IOPS, latency) prediction peak.

[0103] If the load prediction peak is greater than or equal to the preset threshold, and the SSD predicted remaining service life is less than or equal to the set threshold, then the indicator corresponding to the load prediction peak and the SSD remaining service life are the future storage performance bottleneck nodes; thereby achieving accurate positioning of the future storage performance bottleneck nodes and their layers (network layer, protocol layer, storage layer and the hardware devices they rely on).

[0104] (2) Optimizing the performance of the all-flash distributed storage system in advance based on the future storage performance bottleneck node, including:

[0105] Obtain transmission path prediction results. Based on these transmission path prediction results, the predicted remaining lifespan of the SSD, and the predicted peak load, dynamically allocate cold data and hot data to storage nodes of different health levels, allocating future high-load requests to the idle resource pool in advance. Divide the DRAM cache into hot and cold data areas to improve the cache hit rate, achieving collaborative optimization of data layout, cache strategy, and resource allocation.

[0106] And based on the predicted remaining service life of the SSD, the predicted load peak, the performance bottleneck node under the current anomaly and the future storage performance bottleneck node, the automated fault handling process is triggered in a hierarchical manner, and a prediction-driven maintenance plan is built.

[0107] Obtaining transmission path prediction results involves building a topology map based on the storage node status of the all-flash distributed storage system, using machine learning to warn of the impact of future topology changes on the transmission path, and predicting potential bandwidth bottlenecks. Specifically, this includes:

[0108] 1) The internal communication network of the all-flash distributed storage system is abstracted into a topology graph structure G(V,E), where the storage node set V includes active nodes, failed nodes, and idle nodes; the edge set E represents the link bandwidth;

[0109] 2) Based on dynamic programming (dynamic programming is a combination of incremental algorithms for distributed storage system topology optimization, including the Bellman-Ford incremental algorithm for fast rerouting in the event of a single point of failure and memoized search on the DAG to handle multi-path dependencies), the physical and logical connection relationships between storage nodes are obtained. When storage nodes join or leave, or when links are blocked, the topology map is updated in real time.

[0110] 3) Collect historical topology change data (such as node failure frequency, bandwidth fluctuations, and traffic distribution), as well as corresponding transmission path impacts (such as increased latency and decreased throughput). Extract corresponding timing features (node movement trajectory, bandwidth change trend) and topology features (such as the hierarchical depth of the tree structure) from the historical topology change data and the corresponding transmission path impacts. Input the timing features and topology features into the timing model (the timing model is trained using the LSTM model). Based on the timing features and topology features, predict the time of topology changes such as storage node location changes and link disconnections that may occur in the future.

[0111] 4) Based on the timing characteristics, topology characteristics, and topology change time, combined with Q-learning, the potential impact of topology changes on the transmission path is simulated to obtain a transmission path prediction result (i.e., the prediction result of the potential impact of topology changes on the transmission path), such as whether the addition of a storage node will cause root node bandwidth overload.

[0112] Based on the predicted remaining service life of the SSD, the predicted load peak, the performance bottleneck node under the current abnormality, and the future storage performance bottleneck node, the automated fault handling process is triggered in a hierarchical manner, including: ① for low risk (i.e., the predicted remaining service life of the SSD is greater than 20%, the predicted load peak does not exceed the set threshold, and no obvious performance bottleneck is detected, i.e., there are no performance bottleneck nodes at present or in the future), log recording and continuous monitoring; ② for medium risk (the predicted remaining service life of the SSD is between 10% and 20%, the predicted load peak is close to the set threshold, and a minor performance bottleneck is detected, i.e., there are a small number of performance bottleneck nodes at present or in the future, which have not yet affected the business), triggering an early warning notification and optimizing the business load distribution; ③ for high risk (the predicted remaining service life of the SSD is less than 10% or the P / E cycle of the SSD exceeds the limit, the predicted load peak exceeds the set threshold, and a serious performance bottleneck is detected, i.e., there are a large number of performance bottleneck nodes at present or in the future), automatically starting data migration to a spare disk. For example, when it is monitored that the P / E cycle of the SSD exceeds the limit, data migration is automatically triggered, and data is automatically migrated to a healthy node to avoid business interruption.

[0113] Based on the predicted remaining service life of the SSD, a prediction-driven maintenance plan is formed, including: ① Automatically generating an SSD replacement priority list based on the predicted remaining service life of the SSD; ② Based on the SSD replacement priority list, triggering preventive maintenance (such as firmware upgrades and bad block scans) during low-load windows.

[0114] Example 2

[0115] like Figure 2As shown, an embodiment of the present invention provides a cross-layer intelligent location system for storage performance bottlenecks in an all-flash storage system, including a data collection and cross-layer monitoring module, an analysis and decision module, a dynamic policy execution and optimization module, and an AI-driven closed-loop tuning module;

[0116] (1) Data collection and cross-layer monitoring module;

[0117] The data collection and cross-layer monitoring module is used to: monitor and collect cross-layer performance indicator data in real time, perform indicator correlation analysis based on the cross-layer performance indicator data to obtain correlation results between performance indicators of each layer; establish a hierarchical structure and call chain; and construct a cross-layer performance correlation view based on the hierarchical structure and call chain and the correlation results between performance indicators of each layer;

[0118] The data collection and cross-layer monitoring module includes a cross-layer state perception submodule, a distributed tracing probe, and a unified indicator platform; wherein the cross-layer state perception submodule is used to monitor and collect cross-layer performance indicator data in real time;

[0119] The distributed tracing probe is used to: collect FTL (Flash Translation Layer) metadata, perform I / O path tracing based on the FTL metadata, and record I / O path tracing data; establish a hierarchical structure and call chain based on the I / O path tracing data;

[0120] The unified indicator platform is used to normalize the cross-layer performance indicator data into a time series, and perform indicator correlation analysis to obtain correlation results between performance indicators of each layer.

[0121] (2) Analysis and decision-making module;

[0122] The analysis and decision module is used to: perform modeling based on the cross-layer performance correlation view to obtain a cross-layer performance correlation model; set a conditional probability table for each indicator node in the cross-layer performance correlation model, and when an abnormality occurs in a performance indicator of a certain layer, calculate the posterior probability of each indicator node under the abnormal condition based on the conditional probability table of each indicator node; traverse the posterior probabilities of all indicator nodes in the cross-layer performance correlation model, and the indicator node with the largest posterior probability is the performance bottleneck node under the current abnormality; and further used to: identify the current load type based on the cross-layer performance indicator data, and obtain the level of the performance bottleneck node based on the performance bottleneck node under the current abnormality;

[0123] The analysis and decision module includes a cross-layer causal analysis engine and a load classifier; wherein the cross-layer causal analysis engine is used to: perform modeling based on the cross-layer performance correlation view to obtain a cross-layer performance correlation model; when an anomaly occurs, identify the performance bottleneck node under the current anomaly based on the cross-layer performance correlation model; and obtain the level of the performance indicator corresponding to the performance bottleneck node based on the performance bottleneck node under the current anomaly;

[0124] The load classifier is used to identify the current load type based on the cross-layer performance indicator data.

[0125] (3) Dynamic strategy execution and optimization module;

[0126] The dynamic policy execution and optimization module is used to: perform performance optimization based on the level of the performance bottleneck node and the current load type, including: building a load fingerprint library, matching the load fingerprint library with the level of the performance bottleneck node and the current load type to obtain a performance optimization policy; dynamically optimizing and adjusting the current storage policy based on the performance optimization policy; and also used for protocol acceleration and resource scheduling;

[0127] The dynamic policy execution and optimization module includes an adaptive data path controller, a protocol acceleration engine, and a resource scheduler. The adaptive data path controller is used to: build a load fingerprint library, match the load fingerprint library with the level of the performance bottleneck node and the current load type to obtain a performance optimization policy; and dynamically optimize and adjust the current storage policy based on the performance optimization policy.

[0128] The protocol acceleration engine is used to merge multiple I / O requests through a single all-flash distributed storage system call when the performance indicator corresponding to the performance bottleneck node is at the protocol layer, and merge multiple buffers into one atomic operation to achieve protocol acceleration;

[0129] The resource scheduler is used to isolate resources based on the load type and task priority, and reserve independent QoS channels for critical loads; monitor the load (such as IOPS, latency) of the storage nodes of the all-flash distributed storage system in real time. If the load of any storage node exceeds the threshold, the hot data of the storage node is automatically migrated to the idle node; thereby realizing resource scheduling.

[0130] (4) AI-driven closed-loop tuning module;

[0131] The AI-driven closed-loop tuning module is used to collect traffic data of the all-flash distributed storage system and external events that affect the load; predict future storage performance bottleneck nodes based on the traffic data, external events that affect the load and cross-layer performance indicator data; and optimize the performance of the all-flash distributed storage system in advance based on the future storage performance bottleneck nodes.

[0132] The AI-driven closed-loop tuning module includes a health prediction model, a digital twin model, a multi-agent reinforcement learning model, and a self-healing controller. The health prediction model is used to predict the remaining service life of the SSD based on the cross-layer performance indicator data to obtain the SSD predicted remaining service life (RUL). The module also predicts future storage performance bottleneck nodes based on the SSD predicted RUL.

[0133] The digital twin model is used to collect traffic data of the all-flash distributed storage system and external events that affect the load; based on the traffic data and external events that affect the load, the future load peak is predicted to obtain the load (such as IOPS, latency) predicted peak; if the load prediction peak is greater than or equal to a preset threshold, and the SSD predicted remaining service life is less than or equal to the set threshold, then the indicator corresponding to the load prediction peak and the SSD remaining service life are the future storage performance bottleneck node; the digital twin model is used to obtain the transmission path prediction result;

[0134] The multi-agent reinforcement learning model is used to dynamically allocate cold data and hot data to storage nodes of different health levels based on the transmission path prediction results, the predicted remaining service life of the SSD, and the predicted load peak; and to allocate future high-load requests to the idle resource pool in advance;

[0135] The self-healing controller is used to trigger the automated fault handling process in a hierarchical manner and build a prediction-driven maintenance plan based on the SSD's predicted remaining service life, load prediction peak, performance bottleneck nodes under current anomalies, and future storage performance bottleneck nodes.

[0136] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0137] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0138] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0139] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0140] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, ROM, RAM, a magnetic disk, or an optical disk.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A cross-layer intelligent location method for storage performance bottlenecks in an all-flash storage system, characterized in that: The following steps are involved: S100: Real-time monitoring and collection of cross-layer performance indicator data of all-flash distributed storage systems; S200: Establishing the hierarchical structure and call chain of the all-flash distributed storage system; and performing indicator correlation analysis based on the cross-layer performance indicator data to obtain correlation results between performance indicators of each layer; Building a cross-layer performance correlation view based on the hierarchical structure and call chain and the correlation results between the performance indicators of each layer; S300: Modeling is performed based on the cross-layer performance correlation view to obtain a cross-layer performance correlation model; a conditional probability table is set for each indicator node in the cross-layer performance correlation model, and when an abnormality occurs in the performance indicator of a certain layer, the posterior probability of each indicator node under the abnormal condition is calculated based on the conditional probability table of each indicator node; the posterior probabilities of all indicator nodes in the cross-layer performance correlation model are traversed, and the indicator node with the largest posterior probability is the performance bottleneck node under the current abnormality.

2. The method for intelligently locating storage performance bottlenecks across layers according to claim 1, characterized in that: The cross-layer performance indicator data in step S100 includes performance indicator data of the storage layer, network layer, protocol layer and application layer.

3. The method for intelligently locating storage performance bottlenecks across layers according to claim 1, characterized in that: The step S200 includes: Collect FTL metadata, perform I / O path tracing based on the FTL metadata, and record I / O path tracing data; establish a hierarchical structure and call chain of an all-flash distributed storage system based on the I / O path tracing data; wherein the I / O path tracing data includes one or more of the following data: an operation name, timestamp, duration, and general description information of each I / O request.

4. The method for intelligently locating storage performance bottlenecks across layers according to claim 1, characterized in that: The step S200 includes: The timestamp alignment technology is used to timestamp-align the cross-layer performance indicator data. When an abnormality occurs in the performance indicator data of a certain layer, the performance indicator data of other layers at the same timestamp are analyzed to see whether an abnormality occurs. If an abnormality occurs in the performance indicator data of other layers at the same timestamp, the abnormal performance indicator data of the layer is correlated with the abnormal performance indicator data of other layers to obtain the correlation results between the performance indicators of each layer.

5. The method for intelligently locating storage performance bottlenecks across layers according to claim 1, characterized in that: The step S300 includes: The indicators of each layer in the cross-layer performance correlation view are used as indicator nodes, and the correlation relationship between the indicators is used as the dependency relationship. Based on the Bayesian network, the indicator nodes are modeled through directed edge connections to obtain a cross-layer performance correlation model.

6. The method for intelligently locating storage performance bottlenecks across layers according to claim 1, characterized in that: The conditional probability table in step S300 is the probability distribution of each indicator node under different parent node states; The posterior probability of each indicator node under abnormal conditions is calculated using the following formula: Where, P(X i |A=Abnormal) is the performance indicator of a certain layer. When an abnormality occurs, X i The posterior probability of P(A = abnormal|X i ) is X i The probability that a certain layer's performance indicator A will be abnormal when an abnormality occurs; P(X i ) is X i The probability of an anomaly; P(A=anomaly) is the probability of an anomaly occurring in a performance indicator A of a certain layer.

7. The method for intelligently locating storage performance bottlenecks across layers according to claim 1, characterized in that: The step S300 further includes: Based on the cross-layer performance indicator data, the current load type is identified, and based on the performance bottleneck node under the current abnormality, the layer where the performance bottleneck node is located is obtained; and performance optimization is performed based on the layer where the performance bottleneck node is located and the current load type.

8. The method for intelligently locating storage performance bottlenecks across layers according to claim 1, characterized in that: The step S400 is also included: collecting traffic data of the all-flash distributed storage system and external events that affect the load; Predicting future storage performance bottleneck nodes based on the traffic data, external events affecting the load, and the cross-layer performance indicator data; The performance of the all-flash distributed storage system is optimized in advance based on the future storage performance bottleneck node.

9. A cross-layer intelligent location system for storage performance bottlenecks in an all-flash storage system, characterized in that: Including data collection and cross-layer monitoring module and analysis and decision-making module; The data collection and cross-layer monitoring module is used to: monitor and collect cross-layer performance indicator data of the all-flash distributed storage system in real time, perform indicator correlation analysis based on the cross-layer performance indicator data, and obtain correlation results between performance indicators of each layer; and establishing hierarchical structures and call chains; Building a cross-layer performance correlation view based on the hierarchical structure and call chain and the correlation results between the performance indicators of each layer; The analysis and decision module is used to: perform modeling based on the cross-layer performance correlation view to obtain a cross-layer performance correlation model; set a conditional probability table for each indicator node in the cross-layer performance correlation model, and when an abnormality occurs in the performance indicator of a certain layer, calculate the posterior probability of each indicator node under the abnormal condition based on the conditional probability table of each indicator node; traverse the posterior probabilities of all indicator nodes in the cross-layer performance correlation model, and the indicator node with the largest posterior probability is the performance bottleneck node under the current abnormality; and is also used to identify the current load type based on the cross-layer performance indicator data, and obtain the level of the performance bottleneck node based on the performance bottleneck node under the current abnormality.

10. The storage performance bottleneck cross-layer intelligent positioning system according to claim 9, characterized in that: Also includes: Dynamic strategy execution and optimization module and AI-driven closed-loop tuning module; The dynamic strategy execution and optimization module is used to: perform performance optimization based on the level of the performance bottleneck node and the current load type; The AI-driven closed-loop tuning module is used to collect traffic data of the all-flash distributed storage system and external events that affect the load; Predicting future storage performance bottleneck nodes based on the traffic data, external events affecting the load, and the cross-layer performance indicator data; The performance of the all-flash distributed storage system is optimized in advance based on the future storage performance bottleneck node.

Images