Disk fault early warning method, device and equipment based on multi-feature fusion and medium

Abstract

The invention discloses a disk fault early warning method and device based on multi-feature fusion, equipment and a medium, and relates to the field of disk fault prediction.The method comprises the steps that initial operation data is subjected to data preprocessing to obtain processed operation data, and a target sample set is determined based on a preset strategy sample balance method; extracting an initial key feature from the target sample set to determine a target key feature, performing a preset time sequence feature extraction operation on the target sample set to obtain a target time sequence feature, and extracting a target keyword vector from the target sample set to determine a target log text feature; fusing the target key features, the target time sequence features and the target log text features to generate target multi-source features, training an initial fault prediction model by using the target sample set and the target multi-source features, and determining a target fault prediction model; and inputting the monitored real-time operation data into the target fault prediction model to generate a target fault prediction result, and giving an alarm according to the target fault prediction result. Accurate prediction can be carried out before the disk breaks down.

Description

Technical Field

[0001] This invention relates to the field of disk failure prediction, and in particular to a disk failure early warning method, apparatus, device, and medium based on multi-feature fusion. Background Technology

[0002] As a crucial component of data storage systems, the operational status of disks directly impacts data integrity and system reliability. With the continuous expansion of data center scale and the increase in storage density, problems such as data loss, system downtime, and increased maintenance costs caused by disk failures are becoming increasingly prominent.

[0003] Currently, common disk failure prediction methods in the industry mainly fall into several categories, including those based on single-feature thresholds, statistical analysis, and machine learning. Methods based on single-feature thresholds monitor disk SMART (Self-Monitoring, Analysis and Reporting Technology) parameters, determining a failure risk when a parameter exceeds a preset threshold. Methods based on statistical analysis utilize historical operational data to build statistical models, judging disk health status by analyzing the changing trends of SMART features. Methods based on machine learning input multiple SMART features into classification models such as support vector machines, random forests, and neural networks for failure prediction.

[0004] In conclusion, how to accurately predict disk failures is a problem that urgently needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a disk failure early warning method, apparatus, device, and medium based on multi-feature fusion, capable of accurately predicting disk failures before they occur. The specific solution is as follows:

[0006] Firstly, this application provides a disk fault early warning method based on multi-feature fusion, including:

[0007] The initial running data of the target disk is obtained, and the initial running data is preprocessed to obtain processed running data. The target sample set is determined from the processed running data based on a preset strategy sample balancing method.

[0008] Initial key features are extracted from the target sample set, the target correlation coefficient between the initial key features and the preset label is determined, and the target key features are determined based on the target correlation coefficient. The target sample set is subjected to a preset time-series feature extraction operation to obtain the target time-series features. The target keyword vector is extracted from the target sample set through natural language processing, and the target log text features are determined based on the target keyword vector.

[0009] The target key features, the target time-series features, and the target log text features are fused to generate target multi-source features. The target sample set and the target multi-source features are used to train an initial fault prediction model to determine the target fault prediction model.

[0010] The target disk is monitored to obtain real-time operating data, and the real-time operating data is input into the target fault prediction model to generate the target fault prediction result of the target disk. An alarm is issued based on the target fault prediction result.

[0011] Optionally, the preset strategy sample balancing method includes a preset time window expansion strategy, a preset oversampling strategy, and a preset undersampling strategy;

[0012] Accordingly, the sample balancing method based on a preset strategy determines the target sample set from the processed running data, including:

[0013] Identify the historical failure nodes of the target disk, and use a preset time window expansion strategy to determine the target positive sample based on the historical failure nodes;

[0014] The preset oversampling strategy is applied to the target positive sample using a preset synthetic minority class oversampling technique to generate a target synthetic fault sample;

[0015] Integrate the target positive samples with the target synthetic fault samples to determine the initial sample set;

[0016] A preset undersampling strategy is applied to the initial sample set to obtain the target sample set.

[0017] Optionally, determining the key features of the target based on the target correlation coefficient includes:

[0018] Determine whether the target correlation coefficient corresponding to the initial key feature exists within a preset correlation coefficient range;

[0019] If the target correlation coefficient corresponding to the initial key feature exists within a preset correlation coefficient range, then the initial key feature is determined to be a target key feature;

[0020] If the target correlation coefficient corresponding to the initial key feature does not exist in the preset correlation coefficient range, then the initial key feature is removed.

[0021] Optionally, the step of performing a preset temporal feature extraction operation on the target sample set to obtain target temporal features includes:

[0022] Extract sliding window statistics and trend indicators from the monitoring data in the target sample set; wherein, the sliding window statistics include any one or more of the sliding window mean, variance, skewness and kurtosis, and the trend indicators include trend slope and trend acceleration;

[0023] The target time-series characteristics are determined by integrating the sliding window statistics and trend indicators.

[0024] Optionally, the step of fusing the target key features, the target time-series features, and the target log text features to generate target multi-source features includes:

[0025] The target key features, the target time-series features, and the target log text features are fused into an initial feature vector;

[0026] The dimension of the initial feature vector is reduced using a preset dimensionality reduction algorithm to generate target multi-source features.

[0027] Optionally, the step of training an initial fault prediction model using the target sample set and the target multi-source features to determine the target fault prediction model includes:

[0028] Using a pre-defined grouped K-fold cross-validation strategy, the target sample set and the target multi-source features are divided into a training set and a validation set;

[0029] The training set is input into the initial fault prediction model in order to optimize the parameters in the initial fault prediction model and obtain the current trained fault prediction model.

[0030] The validation set is input into the currently trained fault prediction model to obtain the current fault prediction result;

[0031] Determine whether the current fault prediction result is greater than or equal to a preset evaluation index threshold; wherein, the preset evaluation index threshold includes any one or more of the following: recall threshold, precision threshold, and lead time threshold;

[0032] If the current fault prediction result is greater than or equal to the preset evaluation index threshold, then the current trained fault prediction model is determined as the target fault prediction model.

[0033] Optionally, the step of issuing an alarm based on the target fault prediction result includes:

[0034] Determine the prediction score in the target fault prediction result, and determine whether the prediction score meets the preset risk alarm conditions;

[0035] If the predicted score does not meet the preset risk alarm conditions, then the target fault prediction result is marked.

[0036] If the predicted score meets the preset risk alarm conditions, then the preset service alarm system is used to perform a preset alarm convergence operation on the target fault prediction result to determine the target alarm instruction, and an alarm is issued based on the target alarm instruction.

[0037] Secondly, this application provides a disk fault early warning device based on multi-feature fusion, comprising:

[0038] The sample set determination module is used to acquire the initial running data of the target disk, perform data preprocessing on the initial running data to obtain processed running data, and determine the target sample set from the processed running data based on a preset strategy sample balancing method.

[0039] The feature determination module is used to extract initial key features from the target sample set, determine the target correlation coefficient between the initial key features and the preset label, determine the target key features based on the target correlation coefficient, perform a preset time-series feature extraction operation on the target sample set to obtain target time-series features, extract target keyword vectors from the target sample set through natural language processing, and determine target log text features based on the target keyword vectors.

[0040] The model determination module is used to fuse the target key features, the target time series features, and the target log text features to generate target multi-source features, and to train an initial fault prediction model using the target sample set and the target multi-source features to determine the target fault prediction model;

[0041] The fault alarm module is used to monitor the target disk to obtain real-time operating data, input the real-time operating data into the target fault prediction model to generate the target fault prediction result of the target disk, and issue an alarm based on the target fault prediction result.

[0042] Thirdly, this application provides an electronic device, comprising:

[0043] Memory, used to store computer programs;

[0044] A processor is used to execute the computer program to implement the disk fault early warning method based on multi-feature fusion as described above.

[0045] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disk fault early warning method based on multi-feature fusion.

[0046] In summary, this application acquires the initial operating data of the target disk, preprocesses the initial operating data to obtain processed operating data, determines a target sample set from the processed operating data based on a preset strategy sample balancing method, extracts initial key features from the target sample set, determines the target correlation coefficient between the initial key features and preset labels, determines target key features based on the target correlation coefficient, performs a preset time-series feature extraction operation on the target sample set to obtain target time-series features, extracts target keyword vectors from the target sample set through natural language processing, determines target log text features based on the target keyword vectors, fuses the target key features, the target time-series features, and the target log text features to generate target multi-source features, trains an initial fault prediction model using the target sample set and the target multi-source features to determine the target fault prediction model, monitors the target disk to obtain real-time operating data, inputs the real-time operating data into the target fault prediction model to generate the target fault prediction result of the target disk, and issues an alarm based on the target fault prediction result. As described above, this application first obtains the initial operating data of the target disk and preprocesses it to obtain processed operating data. Then, it determines the target sample set based on the preset strategy sample balancing method. Next, it extracts initial key features from the target sample set and determines the target key features by calculating their target correlation coefficient with the preset label. At the same time, it performs a preset time-series feature extraction operation on the target sample set to obtain target time-series features. Then, it extracts target keyword vectors from the target sample set with the help of natural language processing to determine the target log text features. After that, it fuses the target key features, target time-series features and target log text features to generate target multi-source features. It uses the target sample set and target multi-source features to train an initial fault prediction model to determine the target fault prediction model. Finally, it continuously monitors the target disk to obtain real-time operating data, inputs it into the target fault prediction model to generate target fault prediction results, and issues alarms based on the results. In this way, by integrating multiple key features and operational status information, a more comprehensive disk health feature system is constructed, overcoming the problem that traditional methods rely on single or a small number of features, which limits prediction accuracy. At the same time, multiple sample balancing strategies are introduced to alleviate model bias caused by a severe imbalance between the proportion of healthy and faulty disk samples, significantly improving the ability to identify faulty samples. In addition, by combining feature selection and dimensionality reduction methods, highly relevant and high-contribution features are retained, improving model training efficiency and reducing the risk of overfitting, thereby enhancing the stability and interpretability of prediction results. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 This is a flowchart of a disk fault early warning method based on multi-feature fusion disclosed in this application;

[0049] Figure 2 This is a schematic diagram of a specific data acquisition process disclosed in this application;

[0050] Figure 3 This is a schematic diagram of a specific data preprocessing and sample balancing process disclosed in this application;

[0051] Figure 4 This is a schematic diagram of a specific feature fusion process disclosed in this application;

[0052] Figure 5 This is a schematic diagram of a specific model training optimization method disclosed in this application;

[0053] Figure 6 This is a schematic diagram of a specific fault early warning method disclosed in this application;

[0054] Figure 7 This is a flowchart of a specific disk fault early warning method based on multi-feature fusion disclosed in this application;

[0055] Figure 8 This is a schematic diagram of a disk fault early warning device based on multi-feature fusion disclosed in this application;

[0056] Figure 9 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Currently, common disk failure prediction methods in the industry mainly include those based on single-feature thresholds, statistical analysis, and machine learning. Single-feature threshold-based methods monitor disk SMART parameters and determine a failure risk when a parameter exceeds a preset threshold. Statistical analysis-based methods use historical operational data to build statistical models and analyze the changing trends of SMART features to judge the disk's health status. Machine learning-based methods input multiple SMART features into classification models such as support vector machines, random forests, and neural networks for failure prediction. To address the above technical problems, this application discloses a disk failure early warning method, apparatus, device, and medium based on multi-feature fusion, capable of accurately predicting disk failures before they occur.

[0059] See Figure 1 As shown in the figure, this invention discloses a disk fault early warning method based on multi-feature fusion, including:

[0060] Step S11: Obtain the initial running data of the target disk, perform data preprocessing on the initial running data to obtain processed running data, and determine the target sample set from the processed running data based on the preset strategy sample balancing method.

[0061] In this embodiment, as Figure 2 As shown, initial disk operating data is obtained from a multi-source heterogeneous system. This includes, but is not limited to, disk SMART monitoring logs, storage system operating status data, I / O (Input / Output) access statistics, device firmware records, server system logs, and maintenance alarm records. Historical operating data is also obtained through APIs (Application Programming Interfaces) and batch database exports. Message queues such as Kafka and RabbitMQ are used to push real-time disk monitoring data to the data processing pipeline, ensuring latency is below the second level. It's important to note that the collected initial operating data can be indexed and stored by timestamps, unique disk identifiers such as SerialNumber, and acquisition channels, and a distributed storage system is used to ensure data scalability and high availability.

[0062] Understandably, after collecting the initial running data, such as Figure 3As shown, the initial running data needs to be cleaned, aligned, interpolated, and standardized, and sample balancing needs to be performed to address the scarcity of disk failure samples. First, data cleaning and alignment remove duplicate records and outliers from the initial running data, such as negative values ​​or invalid extreme values ​​caused by collection errors. Simultaneously, data sources with different collection frequencies are aligned based on timestamps to ensure that all feature dimensions are synchronized on the same time scale. For missing values, multiple methods are used to impute them, including linear interpolation for short-term missing values ​​and KNN (K-Nearest Neighbor) interpolation based on similar disks for long-term missing values. Next, the initial running data is standardized and normalized. For features with different dimensions, Z-score standardization or Min-Max normalization is used to avoid the impact of dimension differences on model convergence. Periodic features such as time and temperature fluctuations are periodically encoded. Finally, after cleaning, aligning, standardizing, and normalizing the initial running data, the processed running data is obtained.

[0063] Furthermore, in actual disk operation, normal samples account for over 99%, while faulty samples are extremely rare, and the time periods of fault occurrence are unevenly distributed. This causes the model to tend to predict "normal" states during training, ignoring fault signals. Therefore, a preset strategy sample balancing method is used to balance the samples in the processed running data. This preset strategy sample balancing method includes a preset time window expansion strategy, a preset oversampling strategy, and a preset undersampling strategy. First, the historical fault nodes of the target disk can be determined. Then, the preset time window expansion strategy is used to determine target positive samples based on these historical fault nodes. Next, the preset synthetic minority class oversampling technique is used to apply the preset oversampling strategy to the target positive samples to generate target synthetic fault samples. The target positive samples and the target synthetic fault samples are then integrated to determine an initial sample set. Finally, the preset undersampling strategy is applied to the initial sample set to obtain the target sample set. Specifically, a pre-defined time window expansion strategy is used, which marks all data from the N days prior to the failure of the faulty disk as target positive samples, thereby increasing the coverage of positive samples. A pre-defined oversampling strategy is employed, introducing SMOTE (Synthetic Minority Over-sampling Technique) or Borderline-SMOTE to generate new target synthetic fault samples based on feature space interpolation. A pre-defined undersampling strategy is used to perform stratified undersampling of excessive normal samples, ensuring that the diversity of normal data in terms of time and operational status is not destroyed, resulting in the final target sample set. Furthermore, cost-sensitive learning can be used to introduce class weights into the model's loss function, giving higher weight to fault samples in the target sample set during training. This effectively alleviates the training bias problem caused by class imbalance and improves the sensitivity of fault identification.

[0064] Step S12: Extract initial key features from the target sample set, determine the target correlation coefficient between the initial key features and the preset label, determine the target key features based on the target correlation coefficient, perform a preset time-series feature extraction operation on the target sample set to obtain target time-series features, extract target keyword vectors from the target sample set through natural language processing, and determine target log text features based on the target keyword vectors.

[0065] In this embodiment, after balancing the samples, as follows: Figure 4 As shown, multiple feature types need to be integrated to improve the robustness and generalization ability of the model. First, it is determined whether the target correlation coefficient corresponding to the initial key feature exists within a preset correlation coefficient range. If the target correlation coefficient corresponding to the initial key feature exists within the preset correlation coefficient range, the initial key feature is determined to be a target key feature; if the target correlation coefficient corresponding to the initial key feature does not exist within the preset correlation coefficient range, the initial key feature is discarded. Specifically, first, the Pearson correlation coefficient and mutual information between each feature and the target label are calculated. Initial key features are then filtered based on the preset correlation coefficient range. If the target correlation coefficient of the initial key feature is lower than the preset correlation coefficient, the initial key feature is discarded. Subsequently, highly redundant features are removed using the feature correlation matrix. For example, if the correlation coefficient of the initial key feature is higher than 0.9, the corresponding initial key feature is removed to control model complexity and reduce the risk of overfitting. Based on this, tree models such as XGBoost and LightGBM are used to rank the feature importance, retaining the top K most important core features to obtain the target key features.

[0066] Simultaneously, to fully leverage the data information, time-series features were extracted from the continuous monitoring data. This required extracting sliding window statistics and trend indicators from the monitoring data in the target sample set. The sliding window statistics included one or more of the following: mean, variance, skewness, and kurtosis. The trend indicators included trend slope and trend acceleration. These sliding window statistics and trend indicators were integrated to determine the target time-series features. Specifically, sliding window statistics, including mean, variance, skewness, and kurtosis, as well as trend and periodic indicators, were extracted from the continuous monitoring data. Interactive features such as ratios, differences, and products between features were also constructed to capture non-linear relationships, thereby achieving deep fusion of multi-dimensional features and enhancing the model's expressive power. Furthermore, keyword vectors were extracted using NLP (Natural Language Processing), and the target log text features were determined based on these keyword vectors.

[0067] Step S13: Integrate the target key features, the target time-series features, and the target log text features to generate target multi-source features. Use the target sample set and the target multi-source features to train an initial fault prediction model to determine the target fault prediction model.

[0068] In this embodiment, the target key features, the target temporal features, and the target log text features are fused into an initial feature vector. A preset dimensionality reduction algorithm is then used to reduce the dimensionality of the initial feature vector to generate target multi-source features. Specifically, the target key features, target temporal features, and target log text features are fused into a unified feature vector, and dimensionality reduction algorithms such as PCA (principal components analysis) and t-SNE (t-distributed stochastic neighbor embedding) are used to reduce the dimensionality while retaining key information to generate target multi-source features.

[0069] Next, as Figure 5As shown, using a preset grouped K-fold cross-validation strategy, the target sample set and the target multi-source features are divided into a training set and a validation set. The training set is input into the initial fault prediction model to optimize the parameters of the initial fault prediction model, resulting in the current trained fault prediction model. The validation set is input into the current trained fault prediction model to obtain the current fault prediction result. It is then determined whether the current fault prediction result is greater than or equal to a preset evaluation index threshold. The preset evaluation index threshold includes any one or more of the following: recall threshold, precision threshold, and lead time threshold. If the current fault prediction result is greater than or equal to the preset evaluation index threshold, the current trained fault prediction model is determined as the target fault prediction model. Specifically, the balanced samples and fused multi-source features are input into the initial fault prediction model for training, and optimal prediction performance is pursued through parameter optimization. Regarding the selection of the initial fault prediction model, traditional machine learning models such as Random Forest, XGBoost, LightGBM, and CatBoost can be selected based on data characteristics, suitable for small to medium-sized feature sets; or deep learning models such as Transformer can be used to capture long-term dependencies and complex temporal patterns. During training, grouped K-fold cross-validation is employed to ensure that samples from the same device do not appear simultaneously in the training and validation sets, preventing data leakage. Hyperparameter optimization utilizes methods such as Bayesian optimization and grid search to automatically adjust key parameters such as learning rate, regularization coefficient, tree depth, and number of network layers. To suppress overfitting, mechanisms such as L1 / L2 regularization, Dropout, and early stopping can be introduced. After training, the validation set is input into the current trained fault prediction model to obtain the current fault prediction result. Based on the current fault prediction result, the model is evaluated using a comprehensive approach, employing multiple metrics including F1-score, recall, precision, ROC-AUC (Area Under Curve), and lead time. If the current fault prediction result meets all the criteria, the current trained fault prediction model is determined as the target fault prediction model.

[0070] Step S14: Monitor the target disk to obtain real-time operating data, input the real-time operating data into the target fault prediction model to generate the target fault prediction result of the target disk, and issue an alarm based on the target fault prediction result.

[0071] In this embodiment, as Figure 6As shown, a target fault prediction model is deployed using model service frameworks such as TensorFlowServing, TorchServe, and Flask+ONNXRuntime. Real-time runtime data is received through a real-time data stream interface such as KafkaConsumer, and prediction input data is generated according to time windows. This prediction input data is then fed into the target fault prediction model, which outputs the target fault prediction result, supporting millisecond-level response times.

[0072] Furthermore, the prediction score in the target fault prediction result is determined, and it is judged whether the prediction score meets the preset risk alarm conditions. If the prediction score does not meet the preset risk alarm conditions, the target fault prediction result is marked. If the prediction score meets the preset risk alarm conditions, a preset alarm convergence operation is performed on the target fault prediction result using a preset service alarm system to determine the target alarm instruction, and an alarm is issued based on the target alarm instruction. Specifically, the prediction score output by the model is first determined, and a multi-level threshold mechanism is adopted to divide the risk into low, medium, and high risk levels according to the score to support differentiated operation and maintenance responses. If the prediction score indicates that the target disk has a low-risk level fault, the prediction result is marked and recorded in a regular manner. If the prediction score indicates that the target disk has a medium or high-risk level fault, the preset service alarm system is triggered to perform an alarm convergence operation, such as merging continuous high-frequency alarms generated by the same disk into a single event, thereby generating a refined target alarm instruction and implementing the alarm, thereby avoiding alarm storms. Simultaneously, interpretability technologies such as SHAP (Shapley Additive exPlanations, a game theory-based machine learning model interpretation method) and LIME (Local Interpretable Model-agnostic Explanations) are integrated to output the contribution of key features, assisting operations and maintenance personnel in root cause analysis. To ensure the long-term effectiveness of the model, a continuous learning and iteration mechanism is established. By collecting prediction feedback data, the model is periodically retrained, dynamically updating the feature distribution and model parameters to ensure that prediction performance continues to optimize with business development.

[0073] As can be seen from the above, the embodiments of this application first acquire the initial operating data of the target disk and preprocess it to obtain the processed operating data. Then, the target sample set is determined from it according to the preset strategy sample balancing method. Next, the initial key features are extracted from the target sample set, and the target key features are determined by calculating the target correlation coefficient with the preset label. At the same time, the preset time-series feature extraction operation is performed on the target sample set to obtain the target time-series features. The target keyword vector is extracted from the target sample set with the help of natural language processing to determine the target log text features. Then, the target key features, target time-series features and target log text features are fused to generate target multi-source features. The initial fault prediction model is trained using the target sample set and target multi-source features to determine the target fault prediction model. Finally, the target disk is continuously monitored to obtain real-time operating data, which is input into the target fault prediction model to generate the target fault prediction result, and an alarm is issued based on the result. In this way, by integrating multiple key features and operational status information, a more comprehensive disk health feature system is constructed, overcoming the problem that traditional methods rely on single or a small number of features, which limits prediction accuracy. At the same time, multiple sample balancing strategies are introduced to alleviate model bias caused by a severe imbalance between the proportion of healthy and faulty disk samples, significantly improving the ability to identify faulty samples. In addition, by combining feature selection and dimensionality reduction methods, highly relevant and high-contribution features are retained, improving model training efficiency and reducing the risk of overfitting, thereby enhancing the stability and interpretability of prediction results.

[0074] As can be seen from the previous embodiment, this application discloses a disk failure early warning method based on multi-feature fusion, which can accurately predict disk failures before they occur. Next, we will address... Figure 7 The disk fault early warning method based on multi-feature fusion is explained in detail.

[0075] First, historical operational data of the target disk is obtained from the monitoring system, including initial operational data such as SMART attributes, performance metrics, and system logs. This initial operational data undergoes preprocessing operations such as cleaning, noise reduction, and normalization to form well-structured post-processed operational data. Then, considering the scarcity of fault samples, pre-defined strategies such as oversampling or undersampling are used to balance the post-processed operational data, constructing a target sample set capable of supporting effective model learning.

[0076] Next, initial key features reflecting disk health status, such as read / write error rate and reallocation sector count, are extracted from the target sample set. By calculating the correlation coefficient between these initial key features and the preset "fault" label, target key features strongly correlated with faults are selected. To further mine the data value, sliding window analysis is performed on the time-series performance index data to extract target time-series features such as mean and trend slope. Simultaneously, natural language processing technology is used to parse system alarm logs, generating target log text features that characterize abnormal events.

[0077] Finally, key features, temporal features, and textual features are integrated to form a comprehensive multi-source feature set describing the disk status. Using the multi-source feature set and the target sample set, initial fault prediction models such as random forests or LSTM (Long Short-Term Memory) are trained through training strategies such as cross-validation and hyperparameter optimization to obtain a high-precision target fault prediction model.

[0078] During the deployment phase, the target disk is continuously monitored through a real-time data pipeline, collecting real-time operational data and inputting it into the target fault prediction model. The target fault prediction model outputs the probability value of the target disk failing as the prediction result. When the prediction score exceeds a preset risk threshold, the system will automatically trigger an alarm convergence mechanism, generating accurate target alarm commands and notifying maintenance personnel. This achieves a shift from passive fault handling to early proactive warning, effectively ensuring the reliability of the data storage system.

[0079] See Figure 8 As shown, this embodiment of the invention discloses a disk fault early warning device based on multi-feature fusion, comprising:

[0080] The sample set determination module 11 is used to acquire the initial running data of the target disk, perform data preprocessing on the initial running data to obtain processed running data, and determine the target sample set from the processed running data based on a preset strategy sample balancing method.

[0081] The feature determination module 12 is used to extract initial key features from the target sample set, determine the target correlation coefficient between the initial key features and the preset label, determine the target key features based on the target correlation coefficient, perform a preset time-series feature extraction operation on the target sample set to obtain target time-series features, extract target keyword vectors from the target sample set through natural language processing, and determine target log text features based on the target keyword vectors.

[0082] The model determination module 13 is used to fuse the target key features, the target time series features, and the target log text features to generate target multi-source features, and to train an initial fault prediction model using the target sample set and the target multi-source features to determine the target fault prediction model.

[0083] The fault alarm module 14 is used to monitor the target disk to obtain real-time operating data, input the real-time operating data into the target fault prediction model to generate the target fault prediction result of the target disk, and issue an alarm based on the target fault prediction result.

[0084] As described above, this application first obtains the initial operating data of the target disk and preprocesses it to obtain processed operating data. Then, it determines the target sample set based on the preset strategy sample balancing method. Next, it extracts initial key features from the target sample set and determines the target key features by calculating their target correlation coefficient with the preset label. At the same time, it performs a preset time-series feature extraction operation on the target sample set to obtain target time-series features. Then, it extracts target keyword vectors from the target sample set with the help of natural language processing to determine the target log text features. After that, it fuses the target key features, target time-series features and target log text features to generate target multi-source features. It uses the target sample set and target multi-source features to train an initial fault prediction model to determine the target fault prediction model. Finally, it continuously monitors the target disk to obtain real-time operating data, inputs it into the target fault prediction model to generate target fault prediction results, and issues alarms based on the results. In this way, by integrating multiple key features and operational status information, a more comprehensive disk health feature system is constructed, overcoming the problem that traditional methods rely on single or a small number of features, which limits prediction accuracy. At the same time, multiple sample balancing strategies are introduced to alleviate model bias caused by a severe imbalance between the proportion of healthy and faulty disk samples, significantly improving the ability to identify faulty samples. In addition, by combining feature selection and dimensionality reduction methods, highly relevant and high-contribution features are retained, improving model training efficiency and reducing the risk of overfitting, thereby enhancing the stability and interpretability of prediction results.

[0085] In some specific implementations, the preset strategy sample balancing method includes a preset time window expansion strategy, a preset oversampling strategy, and a preset undersampling strategy;

[0086] Accordingly, the sample set determination module 11 may specifically include:

[0087] A positive sample determination unit is used to determine the historical failure nodes of the target disk and determine the target positive sample based on the historical failure nodes using a preset time window expansion strategy.

[0088] The fault sample generation unit is used to apply the preset oversampling strategy to the target positive sample using a preset synthetic minority class oversampling technique to generate a target synthetic fault sample;

[0089] A sample set determination unit is used to integrate the target positive samples and the target synthetic fault samples to determine an initial sample set;

[0090] The sample set acquisition unit is used to perform a preset undersampling strategy on the initial sample set to obtain the target sample set.

[0091] In some specific implementations, the feature determination module 12 may specifically include:

[0092] A coefficient determination unit is used to determine whether the target correlation coefficient corresponding to the initial key feature exists within a preset correlation coefficient interval;

[0093] The first coefficient determination unit is used to determine that the initial key feature is a target key feature if the target correlation coefficient corresponding to the initial key feature exists in a preset correlation coefficient range.

[0094] The second coefficient determination unit is used to remove the initial key feature if the target correlation coefficient corresponding to the initial key feature does not exist in the preset correlation coefficient range.

[0095] In some specific implementations, the feature determination module 12 may specifically include:

[0096] The statistics and indicator extraction unit is used to extract sliding window statistics and trend indicators from the monitoring data in the target sample set; wherein, the sliding window statistics include any one or more of the mean, variance, skewness and kurtosis of the sliding window, and the trend indicators include trend slope and trend acceleration.

[0097] The time series feature determination unit is used to integrate the sliding window statistics and trend indicators to determine the target time series features.

[0098] In some specific implementations, the model determination module 13 may specifically include:

[0099] The feature fusion unit is used to fuse the target key features, the target time-series features, and the target log text features into an initial feature vector;

[0100] The feature generation unit is used to reduce the dimension of the initial feature vector using a preset dimensionality reduction algorithm to generate target multi-source features.

[0101] In some specific implementations, the model determination module 13 may specifically include:

[0102] The feature partitioning unit is used to partition the target sample set and the target multi-source features into a training set and a validation set using a preset grouped K-fold cross-validation strategy.

[0103] The model acquisition unit is used to input the training set into the initial fault prediction model so as to optimize the parameters in the initial fault prediction model and obtain the current trained fault prediction model.

[0104] The result acquisition unit is used to input the validation set into the currently trained fault prediction model to obtain the current fault prediction result.

[0105] The result judgment unit is used to determine whether the current fault prediction result is greater than or equal to a preset evaluation index threshold; wherein, the preset evaluation index threshold includes any one or more of the following: recall threshold, precision threshold, and lead time threshold;

[0106] The model determination unit is used to determine the current trained fault prediction model as the target fault prediction model if the current fault prediction result is greater than or equal to the preset evaluation index threshold.

[0107] In some specific implementations, the fault alarm module 14 may specifically include:

[0108] A score determination unit is used to determine the predicted score in the target fault prediction result and to determine whether the predicted score meets the preset risk alarm conditions.

[0109] The first score determination unit is used to mark the target fault prediction result if the predicted score does not meet the preset risk alarm conditions.

[0110] The second score determination unit is used to perform a preset alarm convergence operation on the target fault prediction result using a preset service alarm system if the predicted score meets the preset risk alarm conditions, so as to determine the target alarm instruction and issue an alarm based on the target alarm instruction.

[0111] Furthermore, embodiments of this application also disclose an electronic device, Figure 9 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0112] Figure 9This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the disk fault early warning method based on multi-feature fusion disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0113] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0114] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0115] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the disk fault warning method based on multi-feature fusion disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0116] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disk fault early warning method based on multi-feature fusion. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0117] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0118] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0120] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0121] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A disk fault early warning method based on multi-feature fusion, characterized in that, include: Acquire the initial running data of the target disk, preprocess the initial running data to obtain processed running data, and determine the target sample set from the processed running data based on the preset strategy sample balancing method; Initial key features are extracted from the target sample set, the target correlation coefficient between the initial key features and the preset label is determined, and the target key features are determined based on the target correlation coefficient. The target sample set is subjected to a preset time-series feature extraction operation to obtain the target time-series features. The target keyword vector is extracted from the target sample set through natural language processing, and the target log text features are determined based on the target keyword vector. The target key features, the target time-series features, and the target log text features are fused to generate target multi-source features. The target sample set and the target multi-source features are used to train an initial fault prediction model to determine the target fault prediction model. The target disk is monitored to obtain real-time operating data, and the real-time operating data is input into the target fault prediction model to generate the target fault prediction result of the target disk. An alarm is issued based on the target fault prediction result.

2. The disk fault early warning method based on multi-feature fusion according to claim 1, characterized in that, The preset strategy sample balancing method includes a preset time window expansion strategy, a preset oversampling strategy, and a preset undersampling strategy; Accordingly, the sample balancing method based on a preset strategy determines the target sample set from the processed running data, including: Identify the historical failure nodes of the target disk, and use a preset time window expansion strategy to determine the target positive sample based on the historical failure nodes; The preset oversampling strategy is applied to the target positive sample using a preset synthetic minority class oversampling technique to generate a target synthetic fault sample; Integrate the target positive samples with the target synthetic fault samples to determine the initial sample set; A preset undersampling strategy is applied to the initial sample set to obtain the target sample set.

3. The disk fault early warning method based on multi-feature fusion according to claim 1, characterized in that, The determination of key features of the target based on the target correlation coefficient includes: Determine whether the target correlation coefficient corresponding to the initial key feature exists within a preset correlation coefficient range; If the target correlation coefficient corresponding to the initial key feature exists within a preset correlation coefficient range, then the initial key feature is determined to be a target key feature; If the target correlation coefficient corresponding to the initial key feature does not exist in the preset correlation coefficient range, then the initial key feature is removed.

4. The disk fault early warning method based on multi-feature fusion according to claim 1, characterized in that, The step of performing a preset temporal feature extraction operation on the target sample set to obtain target temporal features includes: Extract sliding window statistics and trend indicators from the monitoring data in the target sample set; wherein, the sliding window statistics include any one or more of the sliding window mean, variance, skewness and kurtosis, and the trend indicators include trend slope and trend acceleration; The target time-series characteristics are determined by integrating the sliding window statistics and trend indicators.

5. The disk fault early warning method based on multi-feature fusion according to claim 1, characterized in that, The process of fusing the target key features, the target temporal features, and the target log text features to generate target multi-source features includes: The target key features, the target time-series features, and the target log text features are fused into an initial feature vector; The dimension of the initial feature vector is reduced using a preset dimensionality reduction algorithm to generate target multi-source features.

6. The disk fault early warning method based on multi-feature fusion according to claim 1, characterized in that, The step of training an initial fault prediction model using the target sample set and the target multi-source features to determine the target fault prediction model includes: Using a pre-defined grouped K-fold cross-validation strategy, the target sample set and the target multi-source features are divided into a training set and a validation set; The training set is input into the initial fault prediction model in order to optimize the parameters in the initial fault prediction model and obtain the current trained fault prediction model. The validation set is input into the currently trained fault prediction model to obtain the current fault prediction result; Determine whether the current fault prediction result is greater than or equal to a preset evaluation index threshold; wherein, the preset evaluation index threshold includes any one or more of the following: recall threshold, precision threshold, and lead time threshold; If the current fault prediction result is greater than or equal to the preset evaluation index threshold, then the current trained fault prediction model is determined as the target fault prediction model.

7. The disk fault early warning method based on multi-feature fusion according to any one of claims 1 to 6, characterized in that, The step of issuing an alarm based on the target fault prediction result includes: Determine the prediction score in the target fault prediction result, and determine whether the prediction score meets the preset risk alarm conditions; If the predicted score does not meet the preset risk alarm conditions, then the target fault prediction result is marked. If the predicted score meets the preset risk alarm conditions, then the preset service alarm system is used to perform a preset alarm convergence operation on the target fault prediction result to determine the target alarm instruction, and an alarm is issued based on the target alarm instruction.

8. A disk fault early warning device based on multi-feature fusion, characterized in that, include: The sample set determination module is used to acquire the initial running data of the target disk, perform data preprocessing on the initial running data to obtain processed running data, and determine the target sample set from the processed running data based on a preset strategy sample balancing method. The feature determination module is used to extract initial key features from the target sample set, determine the target correlation coefficient between the initial key features and the preset label, determine the target key features based on the target correlation coefficient, perform a preset time-series feature extraction operation on the target sample set to obtain target time-series features, extract target keyword vectors from the target sample set through natural language processing, and determine target log text features based on the target keyword vectors. The model determination module is used to fuse the target key features, the target time series features, and the target log text features to generate target multi-source features, and to train an initial fault prediction model using the target sample set and the target multi-source features to determine the target fault prediction model; The fault alarm module is used to monitor the target disk to obtain real-time operating data, input the real-time operating data into the target fault prediction model to generate the target fault prediction result of the target disk, and issue an alarm based on the target fault prediction result.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the disk fault early warning method based on multi-feature fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the disk fault early warning method based on multi-feature fusion as described in any one of claims 1 to 7.

Images