Slow fault detection method, device, equipment and medium in distributed storage system
By using PCAP and DBSCAN algorithms in Ceph system, combining polynomial regression model and sliding window mechanism, the problem of slow failure detection in distributed storage systems is solved, and the accurate identification and isolation of slow failure events is achieved, which improves the System availability.
Patent Information
- Application Number
- CN202411280438.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2044-09-13
AI Technical Summary
In distributed storage systems, the detection of slow failures is difficult, especially in Ceph systems. Due to its global distribution characteristics and large impact range across fault domains, the existing technology cannot be directly applied to the inference of slow failure states of drives. The root cause of the problem is difficult to accurately identify and isolate.
The PCAP algorithm and DBSCAN algorithm are used to preprocess the data set, and a linear regression model is established through polynomial regression. Combining the sliding window and scoreboard mechanism, slow failure events are identified and counted, scores are calculated to evaluate the severity of the failure, and automatic identification and isolation are achieved. The drive of the problem.
It improves the availability of Ceph system, detects slow failure problems in advance, and effectively improves the availability of the system and long-tail problem solving capabilities.
Smart Images
Figure CN118796133B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method, device, equipment and medium for detecting slow faults in a distributed storage system, and belongs to the technical field of distributed storage fault detection. Background Art
[0002] Slow failures, also known as gray failures, are receiving increasing attention. In a slow failure, a software or hardware component (at runtime) provides less than expected performance. With the advent of new hardware devices and software stacks, the effects of slow failures (e.g., caused by faulty NAND or scheduling issues) are more likely to be noticed, which may have been masked as noise before. Recent studies have shown that the frequency of slow failure events per year may be as frequent as the frequency of stop-and-go events per year.
[0003] Accurately detecting slow failures is challenging. Performance changes caused by internal factors (e.g., SSD garbage collection) or external factors (e.g., workload bursts) can have similar symptoms to slow failures. Unlike stop failures, which are well-defined by standards (e.g., software crashes, data loss), determining slow failures is often empirical in practice and therefore inherently inaccurate. Furthermore, the failures of slow failures are often transient, making it difficult for field engineers to identify, let alone reproduce or reason about the root cause.
[0004] The detection of slow IO in OSD in Ceph distributed storage system mainly tracks the delay exceeding a certain threshold, issues an alarm, and then prompts R&D personnel to intervene and locate the problem. However, there are many reasons for slow requests, and for the type of reasons such as slow failures, it is relatively difficult to capture in the short term. After the problem occurs, it is impossible to track it. The node-based IASO detection framework mainly uses the number of statistical timeouts to calculate the score to obtain the overall score table ranking, and determines the severity of service degradation based on the ranking to decide whether to isolate it from the cluster. However, the current framework cannot be directly applied to Ceph, and for the slow failure status of the drive that cannot be inferred, manual intervention is still required to locate the specific root cause of the problem.
[0005] Inspur Cloud's data centers are spread across the country, and each data center has multiple storage clusters. On top of each cluster, a distributed storage system Ceph or other systems are deployed to support storage services. Each cluster has as few as 40 hard disks and as many as nearly 10,000 hard disks. By default, the data storage drives in each node are generally of the same model. At present, the Ceph system has the ability to detect and isolate faulty hard disks, but when a slow fault occurs, it can only alert a slow request, and it is difficult to track such faults, and the PG in the OSD has the characteristic of being globally distributed across fault domains, and the impact range is very large. Therefore, the present invention proposes a slow fault detection method in a distributed storage system. Summary of the invention
[0006] In order to solve the above problems, the present invention proposes a slow fault detection method, device, equipment and medium in a distributed storage system, which can improve the availability of the Ceph system.
[0007] The technical solution adopted by the present invention to solve the technical problem is:
[0008] In a first aspect, an embodiment of the present invention provides a method for detecting a slow fault in a distributed storage system, comprising the following steps:
[0009] Step S1, collect the data of the interaction between the OC module and all OSDs at intervals of t, calculate the latency and throughput within the time t, and construct a data set;
[0010] Step S2, using PCAP algorithm and DBSCAN algorithm to preprocess the data set;
[0011] Step S3, using polynomial regression to establish a linear regression model;
[0012] Step S4, using a linear regression model to predict the preprocessed data set, identifying out-of-bounds entries according to a slow fault detection threshold and marking them as slow entries;
[0013] Step S5: Use the sliding window and scoreboard mechanism to count all slow failure events and calculate the scores.
[0014] As a possible implementation of this embodiment, in step S1, the collected data includes timestamp, host, disk ID, latency, and throughput.
[0015] As a possible implementation of this embodiment, the step S2, using the PCAP algorithm and the DBSCAN algorithm to preprocess the data set, includes:
[0016] Standardize the data set;
[0017] The DBSCAN algorithm is used to identify outliers in the data set;
[0018] The PCAP algorithm is used to discard outlier entries in the data set.
[0019] As a possible implementation of this embodiment, the step S3, using polynomial regression to establish a linear regression model, includes:
[0020] Construct polynomial features and perform binomial feature conversion to establish a binomial relationship between latency and throughput;
[0021] The binomial relationship between delay and throughput is fitted to construct a linear regression model, which outputs a prediction result of delay.
[0022] As a possible implementation of this embodiment, step S4, using a linear regression model to predict the preprocessed data set, identifying out-of-bounds entries according to a slow fault detection threshold and marking them as slow fault entries, includes:
[0023] Construct a threshold based on the linear regression model: Calculate the standard deviation of the residual between the predicted result of the linear regression model and the actual value; Use the characteristics of the standard normal distribution in statistics to determine the corresponding z quantile with a given confidence level, multiply the standard deviation by the z quantile to obtain the upper limit of the confidence interval, and use the upper limit of the confidence interval as the slow fault detection threshold;
[0024] Identify abnormal entries: After predicting each node, define a sliding window. If the predicted value in the sliding window exceeds the upper limit of the confidence interval, it is considered that a slow failure event has occurred.
[0025] As a possible implementation of this embodiment, step S5 uses a sliding window and scoreboard mechanism to count all slow failure events and calculate scores, including:
[0026] Determine whether a slow fault event occurs;
[0027] If a slow fault event occurs and there was no slow fault event in the previous time period, the score is calculated as The factor decreases, the initial value of Score is 10, if Score continues to decay and is less than 0, it is set to 0;
[0028] If a slow fault event occurs and a slow fault event occurred in the previous time period, the score Score is calculated as The factor increases, and if the score exceeds the threshold, the calculation stops.
[0029] As a possible implementation of this embodiment, the determining whether a slow fault event occurs includes:
[0030] The ratio of slow events to failures within a certain period of time for:
[0031] ,
[0032] Among them, slowCount is the number of slow fault entries, and totalCount is the total number of entries;
[0033] If slowRatio is greater than the slow fault event ratio threshold, it is considered that a slow fault event has occurred.
[0034] In a second aspect, an embodiment of the present invention provides a slow fault detection device in a distributed storage system, comprising:
[0035] The data set construction module is used to collect the data of the interaction between the OC module and all OSDs at intervals of t, calculate the latency and throughput within the time of t, and construct the data set;
[0036] The data set preprocessing module is used to preprocess the data set using the PCAP algorithm and the DBSCAN algorithm;
[0037] A model building module for building a linear regression model using polynomial regression;
[0038] A slow fault detection module, used to predict the preprocessed data set using a linear regression model, identify out-of-bounds entries according to a slow fault detection threshold and mark them as slow entries;
[0039] The slow fault statistics module is used to count all slow fault events and calculate scores using a sliding window and scoreboard mechanism.
[0040] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to perform the steps of the slow fault detection method in any distributed storage system as described above.
[0041] In a fourth aspect, an embodiment of the present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the slow fault detection method in any distributed storage system described above are executed.
[0042] The beneficial effects of the technical solution of the embodiment of the present invention are as follows:
[0043] The present invention is based on capturing disk-level latency and throughput data features, using machine learning technology to establish a regression model, and using a scoreboard mechanism to evaluate the severity of slow failures, timely isolate problematic drives, and improve the availability of the Ceph system. The present invention introduces a new node-level lightweight small data set regression to identify slow failure events at the drive level, discover such problems in advance, and effectively improve the availability of the system and long-tail problems.
[0044] A slow fault detection device in a distributed storage system according to the technical solution of an embodiment of the present invention has the same beneficial effects as a slow fault detection method in a distributed storage system according to the technical solution of an embodiment of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1is a flow chart of a slow failure detection method in a distributed storage system according to an exemplary embodiment;
[0046] Figure 2 is a structural schematic diagram of a slow fault detection device in a distributed storage system according to an exemplary embodiment;
[0047] Figure 3 It is a schematic diagram of the structure of the equipment system involved in the implementation process of the present invention;
[0048] Figure 4 A schematic diagram for visualizing a polynomial regression model;
[0049] Figure 5 This is a schematic diagram of the latency change of disk1 on a host within 1 hour;
[0050] Figure 6 The following is a schematic diagram of the latency variation of disk4 on the same host within 1 hour. DETAILED DESCRIPTION
[0051] In order to more clearly illustrate the technical features of the solution of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.
[0052] In order to detect the slow failure state of the drive in the Ceph system, the present invention uses classic machine learning techniques (PCA, DBSCAN and polynomial regression) to establish a mapping relationship between latency changes and workload pressure; through this mapping relationship, an accurate adaptive threshold can be automatically determined for each node to identify the tracked slow requests. In addition, based on the slow requests, the present invention constructs the corresponding slow failure events and uses a scoreboard mechanism to evaluate the severity of such events.
[0053] like Figure 1 As shown, a method for detecting slow failures in a distributed storage system provided by an embodiment of the present invention includes the following steps:
[0054] Step S1, collect the data of the interaction between the OC module and all OSDs at intervals of t, calculate the latency and throughput within the time t, and construct a data set;
[0055] Step S2, using PCAP algorithm and DBSCAN algorithm to preprocess the data set;
[0056] Step S3, using polynomial regression to establish a linear regression model;
[0057] Step S4, using a linear regression model to predict the preprocessed data set, identifying out-of-bounds entries according to a slow fault detection threshold and marking them as slow entries;
[0058] Step S5: Use the sliding window and scoreboard mechanism to count all slow failure events and calculate the scores.
[0059] As a possible implementation of this embodiment, in step S1, the collected data includes timestamp, host, disk ID, latency, and throughput.
[0060] As a possible implementation of this embodiment, the step S2, using the PCAP algorithm and the DBSCAN algorithm to preprocess the data set, includes:
[0061] Standardize the data set;
[0062] The DBSCAN algorithm is used to identify outliers in the data set;
[0063] The PCAP algorithm is used to discard outlier entries in the data set.
[0064] As a possible implementation of this embodiment, the step S3, using polynomial regression to establish a linear regression model, includes:
[0065] Construct polynomial features and perform binomial feature conversion to establish a binomial relationship between latency and throughput;
[0066] The binomial relationship between delay and throughput is fitted to construct a linear regression model, which outputs a prediction result of delay.
[0067] As a possible implementation of this embodiment, step S4, using a linear regression model to predict the preprocessed data set, identifying out-of-bounds entries according to a slow fault detection threshold and marking them as slow fault entries, includes:
[0068] Construct a threshold based on the linear regression model: Calculate the standard deviation of the residual between the predicted result of the linear regression model and the actual value; Use the characteristics of the standard normal distribution in statistics to determine the corresponding z quantile with a given confidence level, multiply the standard deviation by the z quantile to obtain the upper limit of the confidence interval, and use the upper limit of the confidence interval as the slow fault detection threshold;
[0069] Identify abnormal entries: After predicting each node, define a sliding window. If the predicted value in the sliding window exceeds the upper limit of the confidence interval, it is considered that a slow failure event has occurred.
[0070] As a possible implementation of this embodiment, step S5 uses a sliding window and scoreboard mechanism to count all slow failure events and calculate scores, including:
[0071] Determine whether a slow fault event occurs;
[0072] If a slow fault event occurs and there was no slow fault event in the previous time period, the score is calculated as The factor decreases, the initial value of Score is 10, if Score continues to decay and is less than 0, it is set to 0;
[0073] If a slow fault event occurs and a slow fault event occurred in the previous time period, the score Score is calculated as The factor increases, and if the score exceeds the threshold, the calculation stops.
[0074] As a possible implementation of this embodiment, the determining whether a slow fault event occurs includes:
[0075] The ratio of slow events to failures within a certain period of time for:
[0076] ,
[0077] Among them, slowCount is the number of slow fault entries, and totalCount is the total number of entries;
[0078] If slowRatio is greater than the slow fault event ratio threshold, a slow fault event is considered to have occurred, that is:
[0079] ;
[0080] slowThreadshold is the threshold for the proportion of slow failure events. If curSlow=1, it is considered that a slow failure event has occurred.
[0081] like Figure 2 As shown, an embodiment of the present invention provides a slow fault detection device in a distributed storage system, comprising:
[0082] The data set construction module is used to collect the data of the interaction between the OC module and all OSDs at intervals of t, calculate the latency and throughput within the time of t, and construct the data set;
[0083] The data set preprocessing module is used to preprocess the data set using the PCAP algorithm and the DBSCAN algorithm;
[0084] A model building module for building a linear regression model using polynomial regression;
[0085] A slow fault detection module, used to predict the preprocessed data set using a linear regression model, identify out-of-bounds entries according to a slow fault detection threshold and mark them as slow entries;
[0086] The slow fault statistics module is used to count all slow fault events and calculate scores using a sliding window and scoreboard mechanism.
[0087] like Figure 3 As shown, the devices involved in the implementation of the present invention include OSD (Object-based StorageDevice, object storage device), MGR (MySQL Group Replication) module and OC module (object in Objective-C). The specific process of slow fault detection in the present invention is as follows.
[0088] 1. Data collection.
[0089] The OC module interacts with all OSDs on the node every 15 seconds to collect op_w_latency and op_w_in_bytes, and calculates the throughput and latency during this period. The OC module stops collecting data after collecting data for one hour at idle load every day. The collected data format is as follows:
[0090]
[0091] The collected data includes timestamp, host, disk ID, latency, and throughput.
[0092] Then perform the following steps on the collected dataset.
[0093] 2. PCA and density-based noise application spatial clustering DBSCAN algorithm are used to identify and discard outlier entries.
[0094] 2.1DBSCAN algorithm
[0095] DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a density-based clustering algorithm. DBSCAN defines a cluster as the largest set of density-connected points. It can divide areas with sufficiently high density into clusters and can find clusters of arbitrary shapes in noisy spatial databases.
[0096] DBSCAN clustering definition: the maximum density connected sample set derived from the density reachable relationship, that is, a category of the final clustering, or a cluster. (Note that it is a density connected set). There can be one or more core objects in a cluster. If there is only one core object, all other non-core object samples in the cluster are in this core object. neighborhood; if there are multiple core objects, then any core object in the cluster There must be another core object in the neighborhood, otherwise the two core objects cannot be density-reachable. The set of all samples in the neighborhood forms a DBSCAN cluster.
[0097] So how can we find such a cluster sample set? It randomly selects a core object without a category as a seed, and then finds all the sample sets that can be density-reachable by this core object, which is a cluster. Then continue to select another core object without a category to find a density-reachable sample set, so that another cluster is obtained (such a collection must be density-connected). It keeps running until all core objects have categories.
[0098] 2.2PCA component analysis.
[0099] Data dimensionality reduction is a method for preprocessing high-dimensional feature data. Dimensionality reduction is to retain the most important features of high-dimensional data and remove noise and unimportant features, thereby achieving the purpose of improving data processing speed. In actual production and application, dimensionality reduction can save a lot of time and cost within a certain range of information loss. Dimensionality reduction has also become a very widely used data preprocessing method.
[0100] PCA (Principal Component Analysis), also known as principal component analysis, is the most widely used data dimensionality reduction algorithm. The main idea of PCA is to map n-dimensional features to k-dimensional features. These k-dimensional features are new orthogonal features, also known as principal components, which are k-dimensional features reconstructed on the basis of the original n-dimensional features. The work of PCA is to sequentially find a set of mutually orthogonal coordinate axes from the original space. The selection of new coordinate axes is closely related to the data itself. Among them, the first new coordinate axis is selected in the direction with the largest variance in the original data, the second new coordinate axis is selected in the plane orthogonal to the first coordinate axis so that the variance is the largest, and the third axis is the plane orthogonal to the first and second axes. By analogy, n such coordinate axes can be obtained. In this way, the new coordinate axes obtained find that most of the variance is contained in the first k coordinate axes, and the variance contained in the latter coordinate axes is almost 0. Therefore, the remaining coordinate axes can be ignored, and only the first k coordinate axes containing most of the variance are retained. In fact, this is equivalent to retaining only the dimensional features that contain most of the variance, while ignoring the feature dimensions that contain almost 0 variance, thereby achieving dimensionality reduction of data features.
[0101] 2.3 Data standardization.
[0102] StandardScaler is a commonly used data preprocessing technique used to standardize feature data according to a standard normal distribution with a mean of 0 and a variance of 1. Its main purposes and significance include:
[0103] Eliminate dimensional differences between features: Different features often have different scales and dimensions. For example, the value range of a feature may be between hundreds and thousands, while the value range of another feature may be between 0 and 1. In this case, the dimensional difference between features will affect the model during training, which may cause the model to converge slowly or have poor results. Through standardization, the dimensional difference between features can be eliminated, making it easier for the model to learn the relationship between features.
[0104] Improve model convergence speed and accuracy: In many machine learning algorithms, such as gradient descent, the scale of features directly affects the convergence speed of the algorithm and the final model accuracy. Through standardization, the algorithm can converge faster and obtain a more accurate model.
[0105] Reduce the impact of outliers and outliers: Standardization can transform the data into a normal distribution with a mean of 0 and a variance of 1, which can reduce the impact of outliers and outliers on the model. Because the standardized data has a tighter distribution, the impact of outliers on the mean and variance will be weakened.
[0106] Improve the interpretability of the model: In some cases, if the scales of features vary greatly, the interpretability of model parameters becomes difficult. Through standardization, the size of model parameters can be made comparable, thereby improving the interpretability of the model.
[0107] 2.4 Apply DBSCAN and PCA to filter outliers.
[0108] Before applying the regression model, the necessary preprocessing is to eliminate outlier samples.<latency,throughput> Sample pairs are usually clustered together within a node, but entries from slow failed drives or normal performance variations (such as internal GC) may still be biased. Therefore, before building a polynomial regression model, the present invention first screens out outliers. Using the DBSCAN density-based clustering algorithm (measuring spatial distance) is an effective way to identify potential normal and outliers.
[0109] In short, DBSCAN groups points that are spatially close enough—points whose distance is less than a minimum. Note that data from long-term or permanently slow drives<latency,throughput> The pairs can be clustered together, but far away from the main cluster. Therefore, the present invention only retains the group with the highest score for further modeling.
[0110] Unfortunately, using DBSCAN to filter the original dataset has been shown to be of limited effectiveness. The underlying reason is that throughput and latency are positively correlated. Therefore, the sample (i.e.<latency,throughput> The outliers (i.e., samples from slow failed drives) may be biased toward a specific direction. Therefore, outliers (i.e., samples from slow failed drives) may be incorrectly labeled as internal values. Therefore, the present invention utilizes principal component analysis (PCA) to transform the coordinates and penalize outliers that are perpendicular to the tilt direction to reduce incorrect labeling.
[0111] 2.5 Based on the above invention, the following steps are performed in the project:
[0112] A. First, rely on the pandas library to read data from the CSV file, as shown in Table 1 (Table 1 is a partial data display).
[0113] Table 1: Reading data from a CSV file
[0114]
[0115] B. Log the throughput 10 Get log 10 _throughput, and then construct StandardScaler to standardize the data so that the mean of the data is 0 and the variance is 1; construct a two-dimensional PCA object to perform PCA transformation on the standardized data.
[0116] C. Construct a DBSCAN object with a neighborhood radius of 0.5 and a minimum sample number of 5 to perform prediction and screening on the PCA-converted data.
[0117] D. Delete outliers with predicted values of -1.
[0118] 3. Build a regression model: Perform polynomial regression based on the clean data set to obtain the model.
[0119] 3.1 Model selection
[0120] Since normal drives within a node can have similar latency and throughput relationships (i.e., cluster well together), the present invention can use regression models to describe the behavior of "normal" drives and characterize the range of variation in slow fault detection. Classical regression models include linear, polynomial, and advanced problems such as kernel regression. The present invention does not use linear regression because the dependence of latency on throughput is clearly nonlinear. In addition, advanced models (such as kernel regression) are unnecessary because the mapping of latency to throughput is mainly monotonic (i.e., latency increases with throughput). Polynomial regression is preferred because it can handle nonlinearity while retaining model parsimony (i.e., using enough parameters to achieve the desired goodness of fit).
[0121] 3.2 Polynomial regression.
[0122] Polynomial regression is a regression analysis method that uses polynomial functions to fit data to establish the relationship between independent variables (features) and dependent variables (targets). Unlike linear regression models, polynomial regression models allow polynomial terms for independent variables, making the model more flexible and better able to fit data with nonlinear relationships.
[0123] The principle of the polynomial regression model is as follows:
[0124] A. Model expression: The model expression of polynomial regression can be written as:
[0125] ,
[0126] Among them, y is the dependent variable (target), x is the independent variable (feature), are the coefficients of the model, n is the order of the polynomial, is the error term.
[0127] B. Fitting process: The fitting process of polynomial regression is achieved by minimizing the residual sum of squares, that is, finding the coefficient that minimizes the error between the model prediction value and the true value through an optimization algorithm. This process can be solved using methods such as the least squares method.
[0128] C. Feature transformation: In polynomial regression, if the independent variable has only one feature x, then The polynomial terms with features such as can be obtained by performing polynomial transformation on the original features. For example, for quadratic polynomial regression, the independent variable Convert to , and then fitted using a linear regression model.
[0129] D. In practical applications, it is very important to choose the right polynomial order n. Too low an order may lead to underfitting, where the model cannot fit the complexity of the data well; while too high an order may lead to overfitting, where the model is too sensitive to noise and cannot generalize well to new data.
[0130] E. For the polynomial regression model, various regression model evaluation indicators can be used to evaluate its performance, such as Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Coefficient of Determination (R-squared), etc.
[0131] 3.3 Apply polynomial regression to establish the binomial relationship between latency and log10_throught.
[0132] First, PolynomialFeatures is constructed for binomial feature conversion, then a LinearRegression linear model is constructed for fitting, and finally latency_pred is predicted based on the constructed model.
[0133] 4. Use the predicted upper bound as the slow fault detection threshold. The model is then applied to the original dataset to identify out-of-bounds entries and mark them as slow entries.
[0134] 4.1 Construct a threshold based on the model.
[0135] A. Calculate the standard deviation: Calculate the standard deviation of the residual (prediction error) between the model predicted value 'latency_pred' and the actual value 'latency'. This standard deviation measures the degree of dispersion between the model predicted value and the actual value, that is, the prediction accuracy of the model.
[0136] B. Calculate the upper limit of the confidence interval: Using the characteristics of the standard normal distribution (z distribution) in statistics, the corresponding z quantile is determined by a given confidence level (99.9%). 3.291 is the z quantile at the current 99.9% confidence level. Then multiply the standard deviation by the z quantile to get the upper limit of the confidence interval. This upper limit represents the upper range of the model prediction value at the 99.9% confidence level.
[0137] C. Model visualization, such as Figure 4 shown.
[0138] 4.2 Identify abnormal entries.
[0139] Figure 4The driver latency fitting value (green line) and the 99.9% upper limit (red line) are displayed. The present invention collects statistics for 1 hour per node every day, and after regression per node, collects data every 15 seconds, defines a sliding window of 5 minutes, and detects the proportion of each driver latency exceeding the upper limit every 5 minutes. If the proportion exceeds a certain threshold (configurable here), it is considered that a slow failure event has occurred.
[0140] 5. Identify slow fault events. Use the sliding window and scoreboard mechanism to count the slow fault events of each drive every day, calculate the scores, and report to the Mgr.
[0141] The specific process of calculating the score is:
[0142] Determine whether there is a slow fault event within 2 minutes:
[0143] ,
[0144] Among them, slowCount is the number of slow fault entries analyzed, and totalCount is the total number of entries;
[0145] ,
[0146] slowThreadshold is the slow fault event ratio threshold. If curSlow=1, it is considered that a slow fault event has occurred.
[0147] If a slow fault occurs currently and there is no slow fault event in the previous 2 minutes, the Score is calculated as The factor decreases, and the initial value of Score is 10;
[0148] ;
[0149] ;
[0150] If attenuation continues and the calculated result is less than 0, it is set to 0;
[0151] If a slow fault event occurred within the last 2 minutes, the Score will be Factor increasing:
[0152] ,
[0153] When the score exceeds the threshold (the default value is 100) during calculation, the calculation stops.
[0154] After the calculation is completed, the node driver score is<OSDID,Score> Sent to Mgr, the Mgr module sorts all the drives and sends them to the second-line engineer for intervention.
[0155] The Mgr module collects the node drive scores reported by the OC module every day, gives an overall ranking, and then provides a list of slow fault disks with corresponding proportions according to the configured threshold, and sends it to the second-line engineer for intervention.
[0156] The newly emerged fail-slow faults plague both software and hardware, where the victim components are still running but the performance is degraded. To solve this problem, the present invention introduces a new node-level lightweight small data set regression to identify driver-level slow failure events, detect such problems in advance, and effectively improve the system's availability and long-tail problems.
[0157] Figure 5 This is a schematic diagram of the latency change of disk1 on a host within 1 hour. Figure 6 This is a schematic diagram of the latency variation of disk4 on the same host within 1 hour. Figure 5 and Figure 6 In the figure, the yellow curve is the curve of the original data of the drive, the red curve is the polynomial regression curve after the data of multiple drives in the host are cleaned, and the green curve is the 99.9% upper limit curve of the polynomial regression. Figure 6 It can be clearly seen that no slow failure event has occurred in disk4. Figure 5 There may be a slow failure event on disk1.
[0158] The present invention can identify slow failure events and discover such problems in advance, effectively improving the availability and long-tail problems of the system, and improving the availability of the Ceph system.
[0159] A computer device provided by an embodiment of the present invention includes a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to perform the steps of the slow fault detection method in any distributed storage system as described above.
[0160] Specifically, the above-mentioned memory and processor can be general-purpose memory and processor, which are not specifically limited here. When the processor runs the computer program stored in the memory, it can execute the slow fault detection method in the above-mentioned distributed storage system.
[0161] Those skilled in the art will appreciate that the structure of the computer device does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently.
[0162] In some embodiments, the computer device may also include a touch screen that can be used to display a graphical user interface (e.g., a startup interface of an application) and receive user operations on the graphical user interface (e.g., startup operations on an application). The specific touch screen may include a display panel and a touch panel. The display panel may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. The touch panel may collect the user's contact or non-contact operations on or near it, and generate pre-set operation instructions, for example, the user uses any suitable object such as a finger or stylus to operate on or near the touch panel. In addition, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and posture, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into information that the processor can process, and then sends it to the processor, and can receive and execute commands sent by the processor. In addition, the touch panel can be implemented by various types such as resistive, capacitive, infrared and surface acoustic wave, and any technology developed in the future can also be used to implement the touch panel. Further, the touch panel can cover the display panel, and the user can operate on or near the touch panel covered on the display panel according to the graphical user interface displayed on the display panel. After the touch panel detects the operation on or near it, it is transmitted to the processor to determine the user input, and then the processor provides corresponding visual output on the display panel in response to the user input. In addition, the touch panel and the display panel can be implemented as two independent components or integrated.
[0163] Corresponding to the above-mentioned application startup method, an embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the slow fault detection method in any of the above-mentioned distributed storage systems are executed.
[0164] The application startup device provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present application, its implementation principle and the technical effect produced are the same as those in the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.
[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0166] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0167] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0168] In addition, each functional module in the embodiments provided in the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.
[0169] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0170] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for detecting slow failures in a distributed storage system, characterized in that: The steps include: Step S1, collect the data of the interaction between the OC module and all OSDs at intervals of t, calculate the latency and throughput within the time t, and construct a data set; Step S2, using PCAP algorithm and DBSCAN algorithm to preprocess the data set; Step S3, using polynomial regression to establish a linear regression model; Step S4, using a linear regression model to predict the preprocessed data set, identifying out-of-bounds entries according to a slow fault detection threshold and marking them as slow entries; Step S5, using a sliding window and scoreboard mechanism to count all slow failure events and calculate scores; The step S3, using polynomial regression to establish a linear regression model, includes: Construct polynomial features and perform binomial feature conversion to establish a binomial relationship between latency and throughput; Fitting a binomial relationship between latency and throughput to construct a linear regression model, wherein the linear regression model outputs a prediction result of latency; The step S5 uses a sliding window and scoreboard mechanism to count all slow failure events and calculate scores, including: Determine whether a slow fault event occurs; If a slow fault event occurs and there was no slow fault event in the previous time period, the score is calculated as The factor decreases, the initial value of Score is 10, if Score continues to decay and is less than 0, it is set to 0; If a slow fault event occurs and a slow fault event occurred in the previous time period, the score Score is calculated as The factor increases, and if the score exceeds the threshold, the calculation stops.
2. The method for detecting slow failures in a distributed storage system according to claim 1, characterized in that: In step S1, the collected data includes timestamp, host, disk ID, latency, and throughput.
3. The method for detecting slow failures in a distributed storage system according to claim 1, characterized in that: The step S2, using the PCAP algorithm and the DBSCAN algorithm to preprocess the data set, includes: Standardize the data set; The DBSCAN algorithm is used to identify outliers in the data set; The PCAP algorithm is used to discard outlier entries in the data set.
4. The method for detecting slow failures in a distributed storage system according to claim 1, characterized in that: The step S4, using a linear regression model to predict the preprocessed data set, identifying out-of-bounds entries according to a slow fault detection threshold and marking them as slow fault entries, includes: Construct a threshold based on the linear regression model: Calculate the standard deviation of the residual between the predicted result of the linear regression model and the actual value; Use the characteristics of the standard normal distribution in statistics to determine the corresponding z quantile with a given confidence level, multiply the standard deviation by the z quantile to obtain the upper limit of the confidence interval, and use the upper limit of the confidence interval as the slow fault detection threshold; Identify abnormal entries: After predicting each node, define a sliding window. If the predicted value in the sliding window exceeds the upper limit of the confidence interval, it is considered that a slow failure event has occurred.
5. The method for detecting slow failures in a distributed storage system according to claim 1, characterized in that: The determining whether a slow fault event occurs includes: The ratio of slow events to failures within a certain period of time for: ， Among them, slowCount is the number of slow fault entries, and totalCount is the total number of entries; If slowRatio is greater than the slow fault event ratio threshold, it is considered that a slow fault event has occurred.
6. A slow fault detection device in a distributed storage system, characterized in that: include: The data set construction module is used to collect the data of the interaction between the OC module and all OSDs at intervals of t, calculate the latency and throughput within the time of t, and construct the data set; The data set preprocessing module is used to preprocess the data set using the PCAP algorithm and the DBSCAN algorithm; A model building module for building a linear regression model using polynomial regression; a slow fault detection module, for predicting the preprocessed data set using a linear regression model, identifying out-of-bounds entries according to a slow fault detection threshold and marking them as slow entries; The slow fault statistics module is used to count all slow fault events and calculate scores using a sliding window and scoreboard mechanism; The specific process of the model building module using polynomial regression to build a linear regression model is as follows: Construct polynomial features and perform binomial feature conversion to establish a binomial relationship between latency and throughput; Fitting a binomial relationship between latency and throughput to construct a linear regression model, wherein the linear regression model outputs a prediction result of latency; The specific process of the slow fault statistics module using the sliding window and scoreboard mechanism to count all slow fault events and calculate the scores is as follows: Determine whether a slow fault event occurs; If a slow fault event occurs and there was no slow fault event in the previous time period, the score is calculated as The factor decreases, the initial value of Score is 10, if Score continues to decay and is less than 0, it is set to 0; If a slow fault event occurs and a slow fault event occurred in the previous time period, the score Score is calculated as The factor increases, and if the score exceeds the threshold, the calculation stops.
7. A computer device, characterized in that: It includes a processor, a memory and a bus, the memory stores machine-readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the slow fault detection method in the distributed storage system as described in any one of claims 1-5.
8. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, executes the steps of the slow fault detection method in a distributed storage system as described in any one of claims 1 to 5.