A method, system, and storage medium for detecting slow faults in storage devices.

Abstract

This invention relates to a method, system, and storage medium for detecting slow failures in storage devices in cloud storage systems. The method includes: S1, real-time acquisition of read / write latency and throughput time series of the storage device within an observation period; S2, outlier filtering of the read / write latency and throughput time series; S3, construction of a mapping relationship between throughput and latency, and establishment of a regression model; S4, calculation of the slowness time series based on the regression model; S5, traversal of the slowness event series using a sliding window to form slow events; S6, quantification of slow events into risk scores according to scoring rules, and accumulation based on the time series; S7, determination of whether the accumulated risk score is greater than a pre-configured risk threshold. If so, the corresponding storage device is marked as a slow failure device and isolated offline; otherwise, return to step S1. Compared with existing technologies, this invention has advantages such as accurate location of slow failure devices and strong versatility.

Description

Technical Field

[0001] This invention relates to the field of slow fault detection in storage devices, and in particular to a method, system, and storage medium for slow fault detection in storage devices for cloud storage systems. Background Technology

[0002] Large-scale cloud storage systems aim to provide stable, massive, and high-performance storage services to upstream users. "Large-scale" means maintaining hundreds of large storage clusters and tens of thousands of high-speed storage devices. Therefore, in large-scale scenarios, hardware failures are no longer rare events. In recent years, slow failures, as an emerging failure mode, have become a significant threat to large-scale cloud storage systems. Unlike traditional stop failures (i.e., devices stop working), slow failures cause hardware to be in a state of "still working but with significantly lower-than-expected performance," leading to a significant increase in read / write latency and a significant decrease in read / write throughput. However, the performance degradation caused by slow failures takes many forms: some cause short-term high latency followed by recovery, while others are persistent and recurring over a long period; still others result in continuous slowness lasting for days or even weeks. Regardless of the form, they greatly affect the stability of the storage system. Meanwhile, with the development of hardware technology, high-speed storage devices (such as solid-state drives) are being deployed on a large scale and handling latency-sensitive loads. Therefore, the high latency and low throughput caused by slow failures on high-speed devices have a more significant impact on upstream service quality and user experience. In conclusion, in large-scale cloud storage system scenarios, accurately detecting and promptly isolating slowly failing devices is one of the key challenges in maintaining system stability.

[0003] Slow failure detection is a relatively new topic. In the past three years, leading researchers have primarily proposed solutions that modify the software stack design of distributed system instances, utilizing software-level metrics (such as timeout responses) to detect slow failure nodes. While directly utilizing software-level metrics has achieved good detection results, it has two main drawbacks: 1. Coarse-grained detection: Detection is performed on a node-by-node basis, requiring further (manual) inspection and location of specific faulty devices (SSDs, network cards, CPUs, etc.). 2. Intrusive design: Designed and adapted only for a few distributed instances, with modifications and injections to the corresponding source code. Therefore, it places specific requirements on the instances hosted and deployed on the nodes. However, cloud service providers should avoid touching user-layer code as much as possible. Even within private cloud service facilities, due to the diverse types of cloud services, any changes in upstream business or the need for business expansion necessitate one-to-one source code modifications for different software instances. This creates significant difficulties for system maintenance and iteration. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, and storage medium for detecting slow faults in storage devices in cloud storage systems, thereby improving detection accuracy and accurately locating faulty devices, while also enhancing versatility.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A method for detecting slow failures in storage devices for cloud storage systems includes the following steps:

[0007] S1. Real-time acquisition of read / write latency and throughput time series of storage devices during the observation period;

[0008] S2. Filter outliers from read / write latency and throughput time series;

[0009] S3. Construct a mapping relationship between throughput and latency, and establish a regression model;

[0010] S4. Calculate the time series of the degree of slowness based on the regression model;

[0011] S5. Use a sliding window to traverse the sequence of slow events to form slow events;

[0012] S6. Quantify slow events into risk scores according to the scoring rules, and accumulate them according to the time series.

[0013] S7. Determine whether the cumulative risk score is greater than the pre-configured risk threshold. If so, mark the corresponding storage device as a slow failure device and isolate it offline. Otherwise, return to step S1.

[0014] Step S4 specifically involves: based on the regression model, calculating the upper bound time series of the delay confidence, and dividing the actual delay time series by the upper bound time series of the delay confidence to obtain the time series of the degree of slowness.

[0015] Step S5 specifically involves: using a sliding window to traverse the slowness event sequence; if the median of the slowness level within a certain sliding window is greater than a pre-configured slowness level threshold, then the storage device forms a continuous slow event within that sliding window.

[0016] The scoring rules are established based on the expected detection target. Different score weights are assigned according to the duration and degree of slowness of the storage device within the pre-configured time length, thereby obtaining a risk score.

[0017] A slow failure detection system for storage devices in a cloud storage system, comprising:

[0018] The telemetry data acquisition terminal is used to collect the read / write latency and throughput time series of the storage device in real time during the observation period and send them to the detection terminal;

[0019] The detection end performs the following detection steps: filtering outliers from read / write latency and throughput time series; constructing a mapping relationship between throughput and latency and establishing a regression model; calculating the slowness time series based on the regression model; traversing the slowness event series using a sliding window to form slow events; quantifying slow events into risk scores according to scoring rules and accumulating them based on the time series; determining whether the accumulated risk score is greater than the pre-configured risk threshold. If so, the corresponding storage device is marked as a slow fault device and isolated offline; otherwise, the read / write latency and throughput time series sent by the telemetry data acquisition end are updated and the detection steps are re-executed.

[0020] The calculation of the sluggishness time series based on the regression model is specifically as follows: based on the regression model, calculate the upper confidence time series of the delay, and divide the actual delay time series by the upper confidence time series of the delay to obtain the sluggishness time series.

[0021] The method of using a sliding window to traverse the slow event sequence to form a slow event specifically involves: using a sliding window to traverse the slow event sequence, if the median of the slowness level within a certain sliding window is greater than a pre-configured slowness level threshold, then the storage device forms a continuous slow event in that sliding window.

[0022] The scoring rules are established based on the expected detection target. Different score weights are assigned according to the duration and degree of slowness of the storage device within the pre-configured time length, thereby obtaining a risk score.

[0023] The hardware architecture of the telemetry data acquisition terminal is as follows: multiple homogeneous storage devices form a storage node, multiple storage nodes form a storage cluster, the storage cluster undertakes various upstream cloud services, and the acquisition terminal collects latency and throughput records of the storage devices in each cluster and each node through performance acquisition commands and sends them to the detection terminal.

[0024] A storage medium having a program stored thereon, which, when executed, implements the method described above.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] (1) Fine-grained slow fault detection: This invention can accurately locate the specific equipment with slow faults, respond in a timely manner, and facilitate system operation and maintenance.

[0027] (2) Non-intrusive design: This invention relies entirely on external telemetry data, does not infringe on the user's code space and does not require specifying a particular software version for the running instance, thus maximizing the user's awareness.

[0028] (3) General Technical Model: This invention utilizes the hardware homogeneity and load balancing characteristics of large-scale storage systems to creatively introduce a regression model to establish a mapping relationship between device load pressure (i.e., throughput) and device performance (i.e., latency). Therefore, at the service layer, this invention is applicable to any upper-layer service type; at the device layer, it is applicable to any specification of storage device and mainstream storage device media (e.g., solid-state drives and hard disk drives). It can achieve high accuracy and recall rates on various cloud services (target storage, block storage, etc.) and various storage media and models. At the same time, this invention can also be easily modified to adapt to slow fault detection in other scenarios (e.g., to network cards and routers), demonstrating its versatility.

[0029] (4) Visual representation of slow failure: This invention innovatively designs a set of scoring rules to consider and score the slowness of the equipment from two dimensions: the duration of slowness and the degree of slowness. This helps on-site maintenance personnel to clarify the priorities of slow equipment and effectively maintain the stability of the storage system. Attached Figure Description

[0030] Figure 1 This is a flowchart of the method of the present invention;

[0031] Figure 2 This is a schematic diagram of the system structure of the present invention;

[0032] Figure 3 This is a schematic diagram of a slow fault detection process in one embodiment, where (a) is a read / write delay time series, (b) is a delay confidence upper bound time series, and (c) is a slowness degree time series. Detailed Implementation

[0033] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0034] Example 1

[0035] This embodiment provides a method for detecting slow failures in storage devices for cloud storage systems, such as... Figure 1 As shown, it includes the following steps:

[0036] S1. Real-time acquisition of read / write latency and throughput time series of storage devices during the observation period.

[0037] This embodiment uses a homogeneous storage node (composed of N hard drives of the same specifications and media) as an example. The storage devices within the node are denoted as Disk1, Disk2, ..., Disk... N Let the observation times be T1, T2, ..., T SThe read / write latency (L) and throughput (TP) of each node's storage device are collected during the observation period and denoted as follows: Where i∈{1,…,S} and n∈{1,…,N}, representing Disk n The throughput and latency at time point i.

[0038] Using system performance monitoring commands or scripts, collect the throughput and latency of all hard drives on this node over three hours. Using the average value per minute as a data point, form a time series of length 180, and send the corresponding data to the monitoring terminal.

[0039] S2. Filter outliers from read / write latency and throughput time series.

[0040] The detection end operates on a node-by-node basis, focusing on the "throughput-delay" distribution of nodes within the observation period. Outlier detection is performed. Outliers are those values ​​whose latency is significantly higher than that of most distributions at the same level of throughput.

[0041] S3. Construct a mapping relationship between throughput and latency, and establish a regression model.

[0042] S4. Based on the regression model, calculate the upper bound of the delayed confidence time series. like Figure 3 As shown in (b), the actual delay time series Divide by the upper bound of the delayed confidence time series Obtain the time series of slowness like Figure 3 As shown in (c).

[0043] S5. Use a sliding window to traverse the slow event sequence. Set the sliding window size to 5 data points and calculate the median of the slowness within the sliding window. If the median within a certain sliding window is greater than the slowness threshold 1, then the storage device forms a continuous slow event in that sliding window.

[0044] For example, targeting Figure 3 In Disk1, the median slowness of sliding window 1 is 1.39 (1), while the median slowness of sliding window 2 is 0.89 (<1). Therefore, it is considered that Disk1 experienced a slow event lasting 5 minutes within sliding window 1.

[0045] S6. According to the scoring rules, slow events are quantified into risk scores, and accumulated according to the time series, resulting in a cumulative risk score of 25 for Disk1.

[0046] The scoring rules are established based on the expected detection target. Different score weights are assigned to the duration and degree of slowness of the storage device within a pre-configured time period to obtain a risk score. In one embodiment, if the expected target is to detect a significantly slow storage device, the scoring rules can be formulated to assign additional risk score weights to cases with longer durations and more severe slowness, thereby achieving the detection of the expected target.

[0047] S7. Determine whether the cumulative risk score is greater than the risk threshold 20. If so, mark the corresponding storage device as a slow failure device and isolate it offline; otherwise, return to step S1. In this embodiment, since the cumulative risk score 25 of Disk1 is greater than 20, Disk1 is marked as a slow failure device and isolated offline for subsequent maintenance operations.

[0048] Example 2

[0049] This embodiment provides a slow fault detection system for storage devices in a cloud storage system, such as... Figure 2 As shown, the applicable scenario is a general distributed storage system. The overall hardware architecture follows a hierarchical design of "cluster-rack-node-storage device": a data center has multiple storage clusters; a storage cluster has multiple racks; a rack has multiple storage nodes; and a storage node has multiple devices with the same storage medium and hardware specifications. Simultaneously, at the software level, the storage clusters connect to upstream conventional cloud services (such as block storage and object storage) and are deployed by default with load balancing modules ranging from the cluster level to the node level to ensure load balancing across all hardware levels. Telemetry data is collected through the system's built-in performance monitoring commands and periodically sent to the monitoring end for testing, specifically including:

[0050] The telemetry data acquisition terminal is used to collect the read / write latency and throughput time series of the storage device in real time during the observation period and send them to the detection terminal;

[0051] The detection end performs the following detection steps: outlier filtering on read / write latency and throughput time series; constructing a mapping relationship between throughput and latency, and establishing a regression model; based on the regression model, calculating the latency confidence upper bound time series, dividing the actual latency time series by the latency confidence upper bound time series to obtain the slowness level time series; traversing the slowness level event series using a sliding window, if the median of the slowness level within a certain sliding window is greater than a pre-configured slowness level threshold, then the storage device forms a persistent slow event in that sliding window; quantifying the slow event into a risk score according to the scoring rules, and accumulating it according to the time series; determining whether the accumulated risk score is greater than a pre-configured risk threshold, if so, marking the corresponding storage device as a slow fault device and isolating it offline; otherwise, updating the read / write latency and throughput time series sent by the telemetry data acquisition end and re-executing the detection steps.

[0052] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0053] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for detecting slow failures in storage devices for cloud storage systems, characterized in that, Includes the following steps: S1. Real-time acquisition of read / write latency and throughput time series of storage devices during the observation period; S2. Filter outliers from read / write latency and throughput time series; S3. Construct a mapping relationship between throughput and latency, and establish a regression model; S4. Calculate the time series of sluggishness based on the regression model: Based on the regression model, calculate the upper confidence time series of the delay, and divide the actual delay time series by the upper confidence time series of the delay to obtain the time series of sluggishness. S5. Using a sliding window to traverse the slow event sequence to form a slow event: Using a sliding window to traverse the slow event sequence, if the median of the slowness level in a certain sliding window is greater than the pre-configured slowness level threshold, then the storage device forms a continuous slow event in that sliding window. S6. Quantify slow events into risk scores according to the scoring rules and accumulate them according to the time series. The scoring rules are established based on the expected detection target and assign different score weights to the slow duration and slowness of the storage device within the pre-configured time length, thereby obtaining the risk score. S7. Determine whether the cumulative risk score is greater than the pre-configured risk threshold. If so, mark the corresponding storage device as a slow failure device and isolate it offline. Otherwise, return to step S1.

2. A slow fault detection system for storage devices in a cloud storage system, characterized in that, include: The telemetry data acquisition terminal is used to collect the read / write latency and throughput time series of the storage device in real time during the observation period and send them to the detection terminal; The detection end is used to perform the following detection steps: outlier filtering on read / write latency and throughput time series; constructing a mapping relationship between throughput and latency, and establishing a regression model; The slowness time series is calculated based on a regression model: The upper confidence time series of delay is calculated based on the regression model. The actual delay time series is divided by the upper confidence time series of delay to obtain the slowness time series. A sliding window is used to traverse the slowness event series to form slow events: If the median of the slowness within a certain sliding window is greater than a pre-configured slowness threshold, the storage device forms a persistent slow event within that sliding window. Slow events are quantified into risk scores according to scoring rules, which are accumulated based on the time series. The scoring rules are established based on the expected detection target, assigning different score weights to the duration and degree of slowness within a pre-configured time length for the storage device, thus obtaining a risk score. It is determined whether the accumulated risk score is greater than the pre-configured risk threshold. If so, the corresponding storage device is marked as a slow fault device and isolated offline; otherwise, the read / write latency and throughput time series sent by the telemetry data acquisition terminal are updated, and the detection steps are re-executed.

3. The slow fault detection system for storage devices in a cloud storage system according to claim 2, characterized in that, The hardware architecture of the telemetry data acquisition terminal is as follows: multiple homogeneous storage devices form a storage node, multiple storage nodes form a storage cluster, the storage cluster undertakes various upstream cloud services, and the acquisition terminal collects latency and throughput records of the storage devices in each cluster and each node through performance acquisition commands and sends them to the detection terminal.

4. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in claim 1.

Images