Expanded disk fault prediction method based on multi-source stream mining

Abstract

The invention discloses an extended disk fault prediction method based on multi-source stream mining, which relates to the technical field of disk fault prediction, and comprises the following steps: collecting continuous disk log streams by using a data collector, and carrying out near data preprocessing on the disk log streams; performing random down-sampling on the disk log stream after data preprocessing, and reserving positive samples and related negative samples to obtain training data; and distributing the training data to a base learning device in the prediction model, carrying out training and prediction, and outputting a disk fault prediction result, thereby completing extended disk fault prediction based on multi-source stream mining. According to the method, the problems of extra data movement from data collection to training, data expansion in preprocessing and unbalanced training data distribution in an existing disk fault prediction method are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of disk failure prediction, and in particular to an extended disk failure prediction method based on multi-source flow mining. Background Art

[0002] Ubiquitous disk failures pose a challenge to the reliability of modern data centers. To tolerate disk failures, modern data centers tend to take a proactive approach by predicting impending disk failures. This proactive approach applies machine learning (ML) algorithms to identify disks that are about to fail and replace them in advance. Traditionally, disk failure prediction methods preprocess disk logs to extract numerical values ​​representing disk performance and reliability statistics, including known disk states (i.e., healthy or failed). Using these numerical values ​​as training data, they train prediction models and predict the near-future disk states (e.g., remain healthy or fail). Traditional disk failure prediction methods, whether offline or online learning, are difficult to scale with the amount of data. Offline learning methods assume the availability of all data in advance, while online learning methods treat data as a continuous data stream with partial availability. By aggregating data into a monitoring system (e.g., a distributed storage system or a time series database), both offline and online learning methods treat the set of disks as a whole set. This simplifies the process of obtaining a global view of the data (e.g., global data distribution) to identify failure modes to achieve high prediction accuracy. However, as more and more data is used, obtaining a global view of the data requires a lot of hardware resources. For example, Azure deploys hundreds of millions of disks, and the status data of each disk is recorded every hour. Each record consists of more than 400 statistical attributes. Since the status data is represented by 64-bit floating point values, loading all disk log data in one day to get a global view requires about 6.98TB of memory. Therefore, extended disk failure prediction is critical for large-scale disk deployment.

[0003] However, expanding disk failure prediction faces the following challenges in terms of scalability of pre-processing before training: (1) Additional data movement from data collection to training. Data is collected into monitoring systems. Most existing monitoring systems provide statistical data analysis but lack the specialized features required for disk failure prediction. In addition, existing distributed ML systems are mainly used to optimize ML job execution, with limited consideration for data collection. Due to the use of external ML-based platforms, handling data collection and training separately may incur additional overhead; (2) Data expansion in preprocessing. Traditional preprocessing methods expand the original data through online labeling and feature generation, thereby increasing the demand for memory capacity; (3) Imbalanced distribution of training data. Current incremental learning algorithms, such as bagging (BA) with adaptive sliding windows and adaptive random forest (ARF), are designed for a single continuous data stream and rely on Poisson sampling to distribute training data to base learners. At the same time, there is a data imbalance problem in disk failure prediction. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides an extended disk failure prediction method based on multi-source stream mining, which solves the problems of additional data movement from data collection to training, data expansion in preprocessing, and unbalanced distribution of training data in the existing disk failure prediction methods.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: an extended disk failure prediction method based on multi-source flow mining, comprising the following steps: S1: Use the data collector to collect continuous disk log streams and perform near data preprocessing on the disk log streams; S2: Randomly downsample the disk log stream after recent data preprocessing, retain positive samples and related negative samples, and obtain training data; S3: Assign the training data to the base learner in the prediction model for training and prediction, and output the disk failure prediction results to complete the extended disk failure prediction based on multi-source stream mining.

[0006] Furthermore, in S1, a data collector is used to collect continuous disk log streams. Specifically, the data collector is connected to multiple machines through a cabinet top switch, each machine is equipped with multiple disks, and the data collector is used to regularly collect disk log streams from the connected machines through sufficient network bandwidth of the cabinet top switch.

[0007] Furthermore, the data preprocessing in S1 includes feature extraction, caching, online labeling and time window downsampling.

[0008] Furthermore, random downsampling is performed in S2, specifically: selecting the nearest After collecting negative samples for 1 day, each data collector randomly selects a subset of negative samples from them.

[0009] Furthermore, the step of allocating the training data to the base learners in S3 includes the following sub-steps: S31: using the coordinator to aggregate the local counts of positive samples and related negative samples forwarded from the data collector, and obtain the ratio of global positive samples and related negative samples; S32: using the coordinator to transmit the ratio of global positive samples and relevant negative samples to the data collector; S33: using the data collector to perform Poisson sampling based on the ratio of global positive samples and relevant negative samples, and transmitting the training data after Poisson sampling to the receiver; S34: Use the receiver to perform sequential data retrieval and transmit the retrieval results to the base learner.

[0010] Furthermore, the Poisson sampling configuration in S33 has hyperparameters and , respectively represent the Poisson distribution of the relevant negative samples and the Poisson distribution of the positive samples. The weights are generated by the Poisson distribution of the relevant negative samples and the Poisson distribution of the positive samples, respectively, which represent the update frequency of the relevant negative samples and positive samples during training.

[0011] Furthermore, the S33 includes the following sub-steps: S331: Using the data collector to use the ratio of global positive samples and related negative samples renew ; S332: Based on the update result, the weight of each base learner is calculated using the Poisson distribution of the relevant negative samples and the Poisson distribution of the positive samples, and the weight vectors between the base learners are connected with the feature vector of each sample to form a matrix; S333: Using the data collector to transmit the matrix to the receiver.

[0012] The beneficial effects of the present invention are: (1) By preprocessing the data locally, the need to transfer the raw data to a remote preprocessing location is minimized; (2) To alleviate the data expansion problem, the present invention avoids transferring unused negative samples to the base learner after preprocessing. It performs random downsampling on each data source to ensure that all positive samples and only relevant negative samples are assigned to the base learner; (3) Instead of aggregating all training data into a single stream for distribution, the present invention performs training data distribution on each data source, leveraging lightweight global information. Each data source independently distributes training data to a base learner based on sample weights to minimize data transfer. In addition, when aggregating multiple data streams into a base learner, the present invention maintains the temporal order of time series samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flow chart of an extended disk failure prediction method based on multi-source flow mining.

[0014] Figure 2 This is an architecture diagram of an extended disk failure prediction system based on multi-source stream mining.

[0015] Figure 3Schematic diagram for optimizing data transmission. DETAILED DESCRIPTION

[0016] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0017] like Figure 1 and Figure 2 As shown, an extended disk failure prediction method based on multi-source flow mining includes the following steps: S1: Use the data collector to collect continuous disk log streams and perform near data preprocessing on the disk log streams; S2: Randomly downsample the disk log stream after recent data preprocessing, retain positive samples and related negative samples, and obtain training data; S3: Assign the training data to the base learner in the prediction model for training and prediction, and output the disk failure prediction results to complete the extended disk failure prediction based on multi-source stream mining.

[0018] This paper proposes a general framework ScaleDFP for extending disk failure prediction through multi-source stream mining. ScaleDFP extends disk failure prediction to the problem of multi-source stream mining. By tracing the disk logs originating from multiple data sources, it regards the disk logs as multiple evolving time series data streams. ScaleDFP focuses on the scalability of pre-training preprocessing, including: near data preprocessing, preprocessing the original disk logs near the data source to minimize data transmission to the remote preprocessing facility; random downsampling, retaining all positive samples and related negative samples for training samples to alleviate data expansion in preprocessing; training data distribution, ensuring that the training data from multiple data sources are correctly distributed to the base learner set in the prediction model and optimizing data transmission.

[0019] To perform stream mining, the present invention adopts an incremental learning algorithm as a prediction model, which performs real-time training and prediction on the data stream and makes incremental adjustments based on the incoming labeled samples. The state-of-the-art incremental learning algorithms based on decision tree technology are selected, especially ensemble learning algorithms, which use a single decision tree as a base learner. Such algorithms combine the prediction results of multiple decision trees to improve the overall prediction accuracy.

[0020] The goal of this scheme is to extend disk failure prediction by abstracting it into a multi-source stream mining problem. This scheme conceptualizes disk logs as data streams continuously collected from multiple data sources, and the total number of data sources is denoted by n. Specifically, this scheme considers a typical data center consisting of multiple racks, each rack is connected to multiple machines through a top-of-rack switch, and each machine is equipped with multiple disks. Each rack is regarded as a data source, which generates a data stream consisting of disk logs from the disks connected to the rack.

[0021] In S1, a data collector is used to collect continuous disk log streams. Specifically, the data collector is connected to multiple machines through a cabinet top switch, each machine is equipped with multiple disks, and the data collector is used to regularly collect disk log streams from the connected machines through sufficient network bandwidth of the cabinet top switch.

[0022] The data preprocessing in S1 includes feature extraction, caching, online labeling and time window downsampling.

[0023] When processing a continuous daily disk log stream as input, ScaleDFP extracts the disk logs as features for training and prediction. This solution uses all features by default because it is difficult to select effective features without a global view of all data streams; In order to dynamically label samples, ScaleDFP borrows the idea of ​​caching. It also uses a sliding window (called a cache window) of a fixed size (e.g., 30 days) to cache recent samples for online labeling; ScaleDFP performs online tagging to alleviate data imbalance. Specifically, it performs online tagging in the cache window to mark disks that are about to fail as positive samples. The number of days of additional tagging before the failure is called the additional tagging days. If a failure is found, ScaleDFP will mark the samples within the additional tagging days (e.g., 20 days) before the failure as positive samples; ScaleDFP performs time window downsampling to further alleviate data imbalance. It selects all positive samples in the cache window and negative samples from the last few days.

[0024] Through the above-mentioned near-data preprocessing, data movement from data collection to preprocessing is minimized and resource contention is alleviated. Specifically, ScaleDFP adopts a localized approach to reduce data transfer in preprocessing. Once each data collector collects disk logs, ScaleDFP performs preprocessing locally on each data collector, including feature extraction, caching samples into a dedicated local cache window for online labeling, and time window downsampling. In addition, near-data preprocessing alleviates resource contention by offloading cache tasks (such as feature extraction and online labeling) from centralized processing units to distributed data collectors, thereby achieving parallel processing.

[0025] The preprocessing workflow includes caching and time window downsampling, which increases the amount of preprocessed data. Since negative samples dominate in the cache window, the amount of preprocessed data depends on the time window downsampling choice. Negative samples.

[0026] The random downsampling is performed in S2, specifically: selecting the nearest After collecting negative samples for 1 day, each data collector randomly selects a subset of negative samples from them.

[0027] ScaleDFP uses random downsampling to avoid transferring unused negative samples to the base learners after preprocessing. Specifically, After 10 days of negative samples, each data collector further randomly selects a subset of negative samples, called the downsampling rate (currently set to 10%), ensuring that all positive samples and only relevant negative samples are assigned to the base learner to alleviate the data expansion problem.

[0028] The step S3 allocates the training data to the base learners, including the following steps: S31: using the coordinator to aggregate the local counts of positive samples and related negative samples forwarded from the data collector, and obtain the ratio of global positive samples and related negative samples; S32: using the coordinator to transmit the ratio of global positive samples and relevant negative samples to the data collector; S33: using the data collector to perform Poisson sampling based on the ratio of global positive samples and relevant negative samples, and transmitting the training data after Poisson sampling to the receiver; S34: Use the receiver to perform sequential data retrieval and transmit the retrieval results to the base learner.

[0029] The Poisson sampling configuration in S33 has hyperparameters and , respectively represent the Poisson distribution of the relevant negative samples and the Poisson distribution of the positive samples. The weights are generated by the Poisson distribution of the relevant negative samples and the Poisson distribution of the positive samples, respectively, which represent the update frequency of the relevant negative samples and positive samples during training.

[0030] In order to solve the problem of aggregating training data into a single stream in existing methods, the present invention extends customized Poisson sampling from a single data source in an ensemble learning algorithm to multiple data sources. Set to less than , to ensure that positive samples have greater weight than negative samples.

[0031] This solution also introduces a coordinator in ScaleDFP to aggregate the global counts of positive and negative samples. The coordinator collects local counts of positive and negative samples from all data collectors every day. Since the sample count of each data collector consists of only two integer values, the latency associated with receiving local counts is negligible. For synchronization purposes, the coordinator waits until it receives local counts from all data collectors. The coordinator aggregates the total counts of positive and negative samples separately. Then, it calculates the ratio and will Send back to each data collector. If the number of positive samples is zero, we will Set to a small value (such as 0.001) to prevent the data collector from Also, since aggregating local counts is lightweight, the coordinator does not become a bottleneck for a large number of data collectors.

[0032] The S33 includes the following sub-steps: S331: Using the data collector to use the ratio of global positive samples and related negative samples renew ; S332: Based on the update result, the weight of each base learner is calculated using the Poisson distribution of the relevant negative samples and the Poisson distribution of the positive samples, and the weight vectors between the base learners are connected with the feature vector of each sample to form a matrix; S333: Using the data collector to transmit the matrix to the receiver.

[0033] This scheme optimizes data transfer in ScaleDFP by ensuring that each base learner receives all positive samples and only relevant negative samples. First, the data collector uses renew Given an initial (expressed as ), each data collector will and Update by multiplying . Second, for each sample (positive or negative), each data collector uses the corresponding Poisson distribution ( or ) calculates weights for each base learner. The sequence of weights between base learners is called a weight vector, which is concatenated with the feature vector of each sample. Figure 3 Such feature and weight vectors showing that they come from multiple samples are aggregated into a matrix whose dimensions are determined by the number of samples (rows) and the total number of features, weights, and labels (columns). Third, given that each base learner is deployed on a server, each data collector sends only a subset of the matrix to each base learner. For each base learner, each data collector checks the corresponding weight column, filters out samples with zero weight, and transmits the remaining samples along with their feature vectors and corresponding weight columns.

[0034] ScaleDFP uses receivers to maintain the temporal order of training data received from data collectors. Specifically, each receiver is responsible for receiving The relevant training data of the base learner includes column features (including a label), and Column weights. represents the number of base learners, assuming that the base learners are deployed in servers, of which Can be To achieve synchronization, after receiving training data in a cache window from all data collectors, it sorts all received samples according to the timestamps in the cache window. If there are stragglers in the data collectors, ScaleDFP sets the timeout threshold for waiting for stragglers to a short time (e.g., 2 seconds).

[0035] The extended disk failure prediction process in this solution is decomposed into three stages: Preprocessing: By extracting performance and reliability statistics from disk logs and converting them into a set of numerical values ​​(called feature vectors). For training data, the disk status is converted into numerical values ​​(called labels), while for prediction data, the disk status is unknown. A pair of (feature vector, label) is called a labeled sample. Finally, the training data from multiple streams is distributed to the prediction model.

[0036] Training: Multiple streams of labeled samples are fed into the prediction model simultaneously, and it is continuously updated over time. Specifically, each base learner is trained with a weight vector associated with each sample, bypassing the original Poisson sampling mechanism in ensemble learning. When the receiver supervises a single base learner, all samples have non-zero weights, facilitating direct training input. Similarly, if the receiver supervises multiple base learners, each base learner is trained with samples with non-zero weights, significantly reducing the number of samples compared to no training data allocation. The base learners continuously adapt to concept drift (i.e., the characteristic that the data stream evolves over time) through change detectors. Note that the change detector still needs to use zero-weight samples (only feature and label columns) to detect concept drift.

[0037] Prediction: Use the prediction model to predict the state of the disk in the near future (e.g., the next 30 days). Specifically, each data collector extracts features and forwards unlabeled samples to each receiver in a timely manner without sorting by time because all unlabeled samples come from the same timestamp. Each receiver assigns each unlabeled sample to each base learner for prediction. The prediction model collects and merges the prediction values ​​from the base learners of each sample to obtain the final prediction result.

[0038] In one embodiment of the present invention, a ScaleDFP prototype system is implemented using Python, including a data collector, a coordinator, and a receiver. Network communication is achieved using remote procedure calls (RPCs) through the gRPC library between different components. For the data collector, near data preprocessing is implemented based on the preprocessing workflow in StreamDFP and random downsampling is integrated. After random downsampling, each data collector sends its local positive and negative sample counts to the coordinator. The coordinator then returns the ratio of the total number of positive samples to the total number of negative samples to each data collector. This embodiment also implements customized Poisson sampling on the data collector. Due to the inherent performance limitations of Python in the for loop, the pybind11 library is used here to optimize the Poisson sampling calculation of each sample weight in all base learners. In addition, the data collector transmits the training data to the receiver via RPC. The receiver ensures that the training data is received from all data collectors and sorts the samples by timestamp before writing them to the local file system. The samples are read from the local file system and input into the integrated learning algorithm (i.e., BA and ARF). We implemented an ensemble learning algorithm based on StreamDFP and replaced its original Poisson sampling method with the training data distribution method of this scheme.

[0039] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the invention.

Claims

1. An extended disk failure prediction method based on multi-source flow mining, characterized in that: The following steps are involved: S1: Use the data collector to collect continuous disk log streams and perform near data preprocessing on the disk log streams; S2: Randomly downsample the disk log stream after recent data preprocessing, retain positive samples and related negative samples, and obtain training data; S3: Assign the training data to the base learner in the prediction model for training and prediction, and output the disk failure prediction results to complete the extended disk failure prediction based on multi-source stream mining.

2. The extended disk failure prediction method based on multi-source flow mining according to claim 1 is characterized in that: In S1, a data collector is used to collect continuous disk log streams. Specifically, the data collector is connected to multiple machines through a cabinet top switch, each machine is equipped with multiple disks, and the data collector is used to regularly collect disk log streams from the connected machines through sufficient network bandwidth of the cabinet top switch.

3. The extended disk failure prediction method based on multi-source flow mining according to claim 1 is characterized in that: The data preprocessing in S1 includes feature extraction, caching, online labeling and time window downsampling.

4. The extended disk failure prediction method based on multi-source flow mining according to claim 1 is characterized in that: The random downsampling is performed in S2, specifically: selecting the nearest After collecting negative samples for 1 day, each data collector randomly selects a subset of negative samples from them.

5. The extended disk failure prediction method based on multi-source flow mining according to claim 1 is characterized in that: The step S3 allocates the training data to the base learners, including the following steps: S31: using the coordinator to aggregate the local counts of positive samples and related negative samples forwarded from the data collector, and obtain the ratio of global positive samples and related negative samples; S32: using the coordinator to transmit the ratio of global positive samples and relevant negative samples to the data collector; S33: using the data collector to perform Poisson sampling based on the ratio of global positive samples and relevant negative samples, and transmitting the training data after Poisson sampling to the receiver; S34: Use the receiver to perform sequential data retrieval and transmit the retrieval results to the base learner.

6. The extended disk failure prediction method based on multi-source flow mining according to claim 5 is characterized in that: The Poisson sampling configuration in S33 has hyperparameters and , respectively represent the Poisson distribution of the relevant negative samples and the Poisson distribution of the positive samples. The weights are generated by the Poisson distribution of the relevant negative samples and the Poisson distribution of the positive samples, respectively, which represent the update frequency of the relevant negative samples and positive samples during training.

7. The extended disk failure prediction method based on multi-source flow mining according to claim 6 is characterized in that: The S33 includes the following sub-steps: S331: Using the data collector to use the ratio of global positive samples and related negative samples renew ; S332: Based on the update result, the weight of each base learner is calculated using the Poisson distribution of the relevant negative samples and the Poisson distribution of the positive samples, and the weight vectors between the base learners are connected with the feature vector of each sample to form a matrix; S333: Using the data collector to transmit the matrix to the receiver.

Images