Fault prediction method and system suitable for newly-started type hard disk
By adopting the domain generalization framework and adversarial LSTM automatic encoder model in hard disk failure prediction, the problem of weak generalization capabilities of the newly enabled hard disk failure prediction model is solved, and efficient fault prediction is achieved without historical training data.
Patent Information
- Application Number
- CN202510057946.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-14
- Publication Date
- 2025-04-25
AI Technical Summary
When predicting the failure of the newly enabled hard disk, the existing technology has poor generalization capabilities and insufficient versatility, resulting in the inability to effectively conduct real-time fault prediction without available historical training data.
Adopting the domain generalization framework, an adversarial LSTM autoencoder model is built, and by collecting SMART data from multiple data sources, performing maximum and minimum normalization and data balance, the adversarial LSTM autoencoder model is trained to generate a fault prediction model, and deploy it to the hard disk monitoring system for real-time fault prediction.
Improves the generalization and versatility of the hard disk failure prediction model, allowing it to effectively perform real-time fault prediction of newly enabled hard disks without historical training data.
Smart Images
Figure CN119883767A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of storage security technology, and more specifically, to a fault prediction method and medium suitable for a newly enabled hard disk. Background Art
[0002] In response to the growing demand for Internet services, the construction of data centers has seen a significant growth. The reliability of hardware components is a key factor in ensuring the overall reliability of these IT infrastructures. In data centers, hard disk drives (HDDs) are the main storage solution, but they are prone to failure. HDD failures can lead to data loss, service interruptions, and increased maintenance costs. Therefore, predicting hard disk failures can effectively improve data security, making it an important concern in data centers.
[0003] The most common method of predicting hard drive failure is to use a threshold detection algorithm based on Self-Monitoring, Analysis and Reporting Technology (SMART). This technology is used to monitor and report performance parameters of hard disk drives and solid-state drives. The threshold detection algorithm analyzes the SMART data of the hard drive to determine if it is at risk of failure. When any of the SMART characteristics of the drive exceed the preset threshold, an alarm is issued to notify the user of the potential problem. However, it has been shown that relying solely on SMART's threshold parameters to predict hard drive failure is not a reliable or effective method.
[0004] The effectiveness of hard drive failure prediction depends heavily on smart monitoring data. However, as new hard drives are introduced into the data center during replacement or upgrade, many hard drives from different models and different manufacturers may enter the system. There is no historical data available for model training for these new hard drives. Simply using the existing SMART data in the data center to train the failure prediction model may render the prediction model unusable due to the differences in the distribution of SMART datasets for different models of hard drives. The difference in label information between different models of hard drives poses a challenge to predicting hard drive failures. This difference results in different probability distributions of different hard drive data. However, despite this difference in labeled data, the task of predicting failures remains consistent across different models of hard drives. Therefore, it is urgent to use data from multiple different models of hard drives to train a highly generalizable model that can be used for real-time failure prediction of newly enabled models of hard drives in the data center without available historical training data. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the hard disk fault detection model in the prior art that the generalization ability is not strong and the versatility is not high enough, and to provide a fault prediction method suitable for newly enabled hard disk models.
[0006] To achieve the above object, the technical solution of the present invention is as follows:
[0007] Collect SMART data from multiple data sources;
[0008] Preprocessing the SMART data, wherein the preprocessing includes maximum and minimum normalization and data balancing, and using the preprocessed SMART data as a training data set;
[0009] Based on the domain generalization framework, an adversarial LSTM autoencoder model is constructed, wherein the adversarial LSTM autoencoder model includes: an LSTM autoencoder, a multi-domain maximum average difference distance, and adversarial learning;
[0010] Using the training data set to train the adversarial LSTM autoencoder model to obtain a fault prediction model;
[0011] The fault prediction model is deployed to the hard disk monitoring system of the newly enabled hard disk model and real-time fault prediction is performed.
[0012] Furthermore, the maximum and minimum normalization is specifically: using the maximum and minimum normalization formula to scale the SMART data to between 0 and 1 to eliminate the dimensional difference between different eigenvalues, so that each eigenvalue has the same magnitude.
[0013] Furthermore, the data balancing is performed in an oversampling manner, and sample replication is performed on the faulty hard disk data in the SMART data in the collected hard disk operation data, and the sample replication is performed on the faulty hard disk data in the SMART data in the hard disk operation data, and the sample replication is performed on the healthy hard disk data in the SMART data in the hard disk operation data.
[0014] Furthermore, the source domain of the adversarial LSTM autoencoder model is the preprocessed SMART data, and the target domain of the adversarial LSTM autoencoder model is the SMART data of the target model for which fault prediction is required.
[0015] Furthermore, the LSTM autoencoder is used to capture the temporal information hidden in the SMART data. In the LSTM autoencoder, a long short-term memory network LSTM is used as an encoder Q(x) and a decoder P(z);
[0016] The input SMART data of the LSTM autoencoder is a sequence of input vectors:
[0017] x=x1,x2,...,x T
[0018] Where T is the length of the sequence;
[0019] The encoder Q(x) converts the input sequence into a latent encoded representation:
[0020] z=Q(x)={z1,z2,…,z T}Where z represents the hidden state of LSTM;
[0021] The decoder P(z) maps the latent code to an output sequence:
[0022]
[0023] in is the output of the decoder at time T.
[0024] Furthermore, in the process of training the adversarial long short-term memory network autoencoder model, the LSTM autoencoder is trained to minimize the input vector sequence x and the output sequence The reconstruction error between , the expression of the reconstruction error is:
[0025]
[0026] Here, ‖·‖2 represents the L2 form.
[0027] Furthermore, in order to ensure that the latent code remains unchanged in the input source domain, a regularization term based on the maximum mean difference (MMD) is introduced. The MMD distance is given by:
[0028]
[0029] Among them, Z l and Z t is the latent code of the l domain and t domain from the known probability distribution, n l and n t are the number of samples from the l domain and the t domain respectively, and are the latent codes of the i-th sample from the l-domain and the t-domain, respectively. represents the mapping function that maps the latent code to the Hilbert space (RKHS);
[0030] The average value of the hidden codes from the source domain is replaced by the average value of the latent codes of all visible source domains. MMD is extended to handle multiple domains. The multi-domain maximum average difference distance is used to reduce the distribution difference to generate a feature space containing all visible source domains. The multi-domain maximum average difference distance is defined as follows:
[0031]
[0032] Here, M refers to the number of domains.
[0033] Furthermore, the adversarial learning is used to regularize the distribution of all input data in different domains and reduce overfitting of visible source domain data. The adversarial learning is specifically as follows: introducing an adversarial autoencoder AAE and a prior distribution p(d), and standardizing the obtained potential codes by aligning them with the prior distribution p(d) using an adversarial network. The encoder of the adversarial autoencoder AAE serves as a generator of the adversarial network, and the encoding function of the autoencoder Q(x) matches the prior distribution p(d) with the potential code to generate an aggregated posterior distribution q(z) in the following way:
[0034]
[0035] Where p(x) is the distribution of the input data, and p(Q(x)) is the conditional distribution of the latent code;
[0036] The adversarial network is trained to match the aggregated posterior distribution q(z) to the prior distribution p(d) by minimizing the objective function, which is expressed as:
[0037]
[0038] where D(·) is the discriminator.
[0039] Furthermore, after obtaining the fault prediction model, the following steps are also included:
[0040] A cross-validation method is used to verify the stability and generalization ability of the fault prediction model;
[0041] The fault prediction model is evaluated on multiple data domains, i.e., different models of hard disks, and the performance of the fault prediction model is measured using accuracy, recall, and F1-score to ensure its generalization ability in different domains.
[0042] Another aspect of the present invention provides a fault prediction system applicable to a newly enabled hard disk, characterized in that the fault prediction system applicable to a newly enabled hard disk comprises: a memory and a processor, wherein the memory comprises a program of a fault prediction method applicable to a newly enabled hard disk, and when the program of a fault prediction method applicable to a newly enabled hard disk is executed by the processor, the following steps are implemented:
[0043] Collect SMART data of various hard disk operation data from multiple data sources;
[0044] Preprocessing the SMART data, wherein the preprocessing includes maximum and minimum normalization and data balancing, and using the preprocessed SMART data as a training data set;
[0045] Based on the domain generalization framework, an adversarial LSTM autoencoder model is constructed, wherein the adversarial LSTM autoencoder model includes: an LSTM autoencoder, a multi-domain maximum average difference distance, and adversarial learning;
[0046] Using the training data set to train the adversarial LSTM autoencoder to obtain a fault prediction model;
[0047] The fault prediction model is deployed to the hard disk monitoring system of the newly enabled hard disk model and real-time fault prediction is performed.
[0048] Compared with the prior art, the present invention has the following beneficial effects:
[0049] The present invention constructs an adversarial LSTM autoencoder model through a domain generalization framework, and uses the operating data of various hard disks to train it to obtain a fault prediction model. By using domain generalization technology, the generalization ability of the hard disk fault prediction model is enhanced, and it has high versatility and is suitable for real-time fault prediction of newly enabled hard disk models in data centers. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A schematic diagram of the steps of the method for predicting a hard disk failure of the present application;
[0051] Figure 2 This is a schematic diagram of the structure of the adversarial LSTM autoencoder model of this application;
[0052] Figure 3 Schematic diagram of the domain generalization framework of the adversarial LSTM autoencoder model of this application;
[0053] Figure 4 This is a schematic diagram of the structure of the LSTM autoencoder of this application;
[0054] Figure 5 This is a flowchart of a fault prediction method applicable to a newly enabled hard disk model according to Embodiment 1 of the present application. DETAILED DESCRIPTION
[0055] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0057] Example 1
[0058] See also Figure 1 and Figure 5 , Figure 1 This is a schematic diagram of the steps of the method for predicting a hard disk failure of the present application. Figure 5 This is a flowchart of a fault prediction method applicable to a newly enabled hard disk of an embodiment of the present application. Embodiment 1 of the present application provides a fault prediction method applicable to a newly enabled hard disk of a new model, comprising the following steps:
[0059] S1: Data collection: Collect hard disk operation data from multiple data sources (such as hard disks of different manufacturers and models), including SMART (Self-Monitoring, Analysis and Reporting Technology) attribute data.
[0060] S2: Data preprocessing:
[0061] First, the maximum and minimum normalization formula is used to process each SMART attribute data, and the data is scaled to a fixed range (0-1). The purpose of this is to eliminate the dimensional differences between different feature values, so that each feature has the same magnitude, which is convenient for subsequent deep learning algorithm processing. The maximum and minimum normalization formula is as follows:
[0062]
[0063] where x norm is the normalized value of x, max and x min They are the maximum and minimum values of the SMART attributes, respectively.
[0064] Since the data center records relatively little hard disk failure data, most of which are normal data of healthy hard disks, these data need to be balanced so that the subsequent deep learning algorithm can learn the data characteristics of the faulty hard disk. In the present invention, oversampling is used to balance the data, and the collected faulty hard disk data samples are copied to keep the faulty hard disk data and the healthy hard disk data balanced in quantity.
[0065] S3: Model building:
[0066] Based on the domain generalization framework, an adversarial LSTM autoencoder model is constructed. Domain generalization (DG) is a machine learning method that aims to train a model to perform well on unseen target domains, even if these target domains did not appear during training. See Figure 3 , Figure 3This is a schematic diagram of the domain generalization framework of the adversarial LSTM autoencoder model of this application, using Figure 3 The model is constructed by using the domain generalization framework shown in the figure. The data in the source domain is the existing training data in the data center. Different source domains represent the data of different hard disk models in the data center. The target domain is the SMART data of the target model that needs to be predicted for faults, that is, the SMART data of the newly enabled hard disk model.
[0067] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of the adversarial LSTM autoencoder model of this application. The adversarial LSTM autoencoder model built based on the domain generalization framework includes three main parts: LSTM autoencoder, multi-domain maximum mean difference distance, and adversarial learning. The main goal is to capture the fault characteristics hidden in the timing information of SMART data through the LSTM autoencoder, and generate a feature space containing all visible source domains by reducing the distribution difference using the multi-domain MMD distance. In addition, by introducing the prior distribution of adversarial training, the distribution of all input data in different domains is regularized. The introduction of prior distribution helps to reduce overfitting of visible source domain data, thereby improving the model's ability to generalize unfamiliar data.
[0068] LSTM Autoencoder:
[0069] The SMART data of the hard disk contains timing information. Traditional autoencoders use conventional neural units as encoders and decoders. In order to better capture the timing information of SMART about faults, LSTM is used as the encoder Q(x) and decoder P(z) of the autoencoder. The structure of the LSTM autoencoder is as follows: Figure 4 The input SMART can be represented as a vector sequence x=x1,x2,...,x T , where T is the length of the sequence. The encoder Q(x) converts the input sequence into a latent encoding representation using the following formula:
[0070] z=Q(x)={z1,z2,...,z T}
[0071] Where z represents the hidden state of LSTM.
[0072] The decoder P(z) then maps the latent code to an output sequence:
[0073]
[0074] in is the output of the decoder at time t.
[0075] The LSTM autoencoder is trained to minimize the input sequence x and the output sequence The reconstruction error between:
[0076]
[0077] where ‖·‖2 represents the L2 norm.
[0078] Maximum average difference distance among multiple domains:
[0079] To ensure that the latent code remains unchanged in the input source domain, a maximum mean discrepancy (MMD) based regularization term is introduced, called z m Represents the potential codes of different source domains. Multi-domain MMD is used to measure the differences between the distributions of all source regions. If there are potential codes Z from two domains with known probability distributions, l and Z t , then the MMD distance is given by:
[0080]
[0081] in, and are the latent codes of the i-th sample from the l-domain and the t-domain, respectively. represents the mapping function that maps the latent code to the Hilbert space (RKHS).
[0082] MMD is extended to handle multiple domains by replacing the mean of the latent codes from the source domain with the mean of the latent codes from all visible source domains. The multi-domain MMD distance is defined as follows
[0083]
[0084] Adversarial Learning:
[0085] MMD-based regularization helps learn latent codes and reduce variations across source domains, but it may overfit the source domain, hampering generalization to the target domain. To address this issue, an adversarial autoencoder AAE is adopted and a prior distribution p(d) is introduced. The obtained latent codes are normalized by aligning them with p(d) using an adversarial network. The encoder of the adversarial autoencoder AAE serves as the generator of the adversarial network. The encoding function of the encoder Q(x) of the adversarial autoencoder matches the prior distribution p(d) with the latent code to generate the aggregated posterior distribution q(z) in the following way:
[0086]
[0087] where p(x) is the distribution of the input data and p(Q(x)) is the conditional distribution of the latent encoding vector.
[0088] The adversarial network is trained to match the aggregated posterior distribution q(z) to the prior distribution p(d) by minimizing the following objective function:
[0089]
[0090] The encoder adjusts the aggregated posterior distribution in such a way that the generated aggregated posterior distribution makes it as difficult as possible for the discriminator network to distinguish whether the sample comes from the latent coding distribution or the introduced true prior distribution, thereby preventing overfitting.
[0091] The SMART data from the hard disk carries label information, so it is necessary to include these labels in the training phase. Therefore, a prediction classifier C is integrated into the model. The classifier consists of two Dense layers and uses the cross entropy loss function. as its standard.
[0092] The training objective of the adversarial LSTM autoencoder model can be divided into four different parts: classifier loss, LSTM-autoencoder reconstruction loss, maximum mean difference loss, and adversarial loss. The overall optimization process of the adversarial LSTM autoencoder can be regarded as a minimum optimization problem:
[0093]
[0094] Among them, C represents the parameters of the classifier, and λ0, λ1, and λ2 are hyperparameters that balance the three terms.
[0095]
[0096] Table 1 Network parameters of the adversarial LSTM autoencoder model
[0097] Table 1 lists in detail the network parameters of the adversarial LSTM autoencoder model in this embodiment 1, including the layer Layer used by each module network, the number of nodes Node and the activation function used.
[0098] S4: Model training:
[0099] The preprocessed data in step S2 is used to train the adversarial LSTM autoencoder model. In the model parameter selection, the Adam optimizer is used for training, and the initial learning rate is configured as 1×10 4 , and the batch size and number of times are set to 100. In order to avoid overfitting, the present invention uses the dropout method and sets the dropout value to 0.25. After the training is completed, the fault prediction model is obtained
[0100] S5: Model deployment: Integrate the trained fault prediction model into the hard disk monitoring system.
[0101] S6: Perform real-time fault prediction. When the new model hard disk is running, the real-time monitoring system monitors the operating status of the hard disk and inputs the recorded SMART data into the deployed domain generalization model for fault prediction. If the prediction is healthy, no operation is required. If the prediction is a fault, the maintenance personnel of the data center can perform operations such as data migration and replacement of healthy hard disks to avoid data loss.
[0102] Example 2
[0103] Based on Example 1, this Example 2 adds a verification and evaluation step after the model training is completed in step S4, including:
[0104] The cross-validation method was used to further verify the stability and generalization ability of the model;
[0105] The model is then evaluated on multiple data domains (different models of hard disks), using metrics such as accuracy, recall, and F1-score to measure the performance of the fault prediction model to ensure that its generalization ability in different domains meets expectations.
[0106] The other steps of this embodiment 2 are the same as those of embodiment 1 and will not be repeated here.
[0107] Example 3
[0108] Based on Example 1, this Example 3 also provides a fault prediction system applicable to a newly enabled hard disk model, characterized in that the fault prediction system applicable to a newly enabled hard disk model includes: a memory and a processor, the memory includes a program of a fault prediction method applicable to a newly enabled hard disk model, and when the program of a fault prediction method applicable to a newly enabled hard disk model is executed by the processor, the steps of a fault prediction method applicable to a newly enabled hard disk model as described in Example 1 are implemented.
[0109] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store used or received data, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.
[0110] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A fault prediction method applicable to a newly enabled hard disk, characterized in that: The following steps are involved: Collect SMART data of various hard disk operation data from multiple data sources; Preprocessing the SMART data, wherein the preprocessing includes maximum and minimum normalization and data balancing, and using the preprocessed SMART data as a training data set; Based on the domain generalization framework, an adversarial LSTM autoencoder model is constructed, wherein the adversarial LSTM autoencoder model includes: an LSTM autoencoder, a multi-domain maximum average difference distance, and adversarial learning; Using the training data set to train the adversarial LSTM autoencoder model to obtain a fault prediction model; The fault prediction model is deployed to the hard disk monitoring system of the newly enabled hard disk model and real-time fault prediction is performed.
2. A fault prediction method for a newly-used hard disk according to claim 1, characterized in that: The maximum and minimum normalization is specifically: using the maximum and minimum normalization formula to scale the SMART data to between 0 and 1 to eliminate the dimensional differences between different eigenvalues so that each eigenvalue has the same magnitude.
3. The fault prediction method for a newly-used hard disk according to claim 1, characterized in that: The data balancing is performed in an oversampling manner, and the faulty hard disk data in the SMART data in the collected hard disk operation data is copied to maintain a quantitative balance with the healthy hard disk data in the SMART data in the hard disk operation data.
4. The method for predicting a failure of a newly-used hard disk according to claim 1, characterized in that: The source domain of the adversarial LSTM autoencoder model is the preprocessed SMART data, and the target domain of the adversarial LSTM autoencoder model is the SMART data of the target model for which fault prediction is required.
5. The fault prediction method for a newly-used hard disk according to claim 1, characterized in that: The LSTM autoencoder is used to capture fault features hidden in the time series information of SMART data. In the LSTM autoencoder, a long short-term memory network LSTM is used as an encoder Q(x) and a decoder P(z); The input SMART data of the LSTM autoencoder is a sequence of input vectors: x=x1,x2,...,x T Where T is the length of the sequence; The encoder Q(x) converts the input sequence into a latent encoded representation: Among them, z represents the hidden state of LSTM; The decoder P(z) maps the latent code to an output sequence: in is the output of the decoder at time T.
6. A fault prediction method for a newly-used hard disk according to claim 5, characterized in that: During the training of the adversarial LSTM autoencoder model, the LSTM autoencoder is trained to minimize the input vector sequence x and the output sequence The reconstruction error between , the expression of the reconstruction error is: Here, ‖·‖2 represents the L2 form.
7. A fault prediction method applicable to a newly-used hard disk according to any one of claims 4 or 5, characterized in that: In order to ensure that the latent code remains unchanged in the input source domain, a regularization term based on the maximum mean difference (MMD) is introduced. The MMD distance is given by: Among them, Z l and Z t is the latent code of the l domain and t domain from the known probability distribution, n l and n t are the number of samples from the l domain and the t domain respectively, and are the latent codes of the i-th sample from the l-domain and the t-domain respectively. represents the mapping function that maps the latent code to the Hilbert space RKHS; The average value of the hidden codes from the source domain is replaced by the average value of the latent codes of all visible source domains. MMD is extended to handle multiple domains. The multi-domain maximum average difference distance is used to reduce the distribution difference to generate a feature space containing all visible source domains. The multi-domain maximum average difference distance is defined as follows: Here, M refers to the number of domains.
8. A fault prediction method applicable to a newly-used hard disk according to any one of claims 1 or 5, characterized in that: The adversarial learning is used to regularize the distribution of all input data in different domains and reduce overfitting of visible source domain data. The adversarial learning is specifically as follows: introducing an adversarial autoencoder AAE and a prior distribution p(d), and standardizing the obtained potential codes by aligning them with the prior distribution p(d) using an adversarial network. The encoder of the adversarial autoencoder AAE serves as a generator of the adversarial network, and the encoding function of the autoencoder Q(x) matches the prior distribution p(d) with the potential code to generate an aggregated posterior distribution q(z) in the following way: Where p(x) is the distribution of the input data, and p(Q(x)) is the conditional distribution of the latent code; The adversarial network is trained to match the aggregated posterior distribution q(z) to the prior distribution p(d) by minimizing the objective function, which is expressed as: where D(·) is the discriminator.
9. The method for predicting a failure of a newly-used hard disk according to claim 1, characterized in that: After the fault prediction model is obtained, the following steps are also included: A cross-validation method is used to verify the stability and generalization ability of the fault prediction model; The fault prediction model is evaluated on multiple data domains, i.e., different models of hard disks, and the performance of the fault prediction model is measured using accuracy, recall, and F1-score to ensure that its generalization ability in different domains meets expectations.
10. A fault prediction system for a newly-used hard disk, characterized in that: The fault prediction system for a newly enabled hard disk includes: a memory and a processor, wherein the memory includes a program for a fault prediction method for a newly enabled hard disk, and when the program for a fault prediction method for a newly enabled hard disk is executed by the processor, the following steps are implemented: Collect SMART data of various hard disk operation data from multiple data sources; Preprocessing the SMART data, wherein the preprocessing includes maximum and minimum normalization and data balancing, and using the preprocessed SMART data as a training data set; Based on the domain generalization framework, an adversarial LSTM autoencoder model is constructed, wherein the adversarial LSTM autoencoder model includes: an LSTM autoencoder, a multi-domain maximum average difference distance, and adversarial learning; Using the training data set to train the adversarial LSTM autoencoder to obtain a fault prediction model; The fault prediction model is deployed to the hard disk monitoring system of the newly enabled hard disk model and real-time fault prediction is performed.