Storage fault prediction method based on adversarial generation and deep learning

Abstract

The invention relates to a storage fault prediction method based on adversarial generation and deep learning, and the method comprises the steps: collecting historical health state data and historical fault data of a storage system, carrying out the end-to-end automatic feature extraction after preprocessing, and obtaining a real fault feature; generating an adversarial sample, inputting the adversarial sample into the CNN to extract adversarial features, and fusing the real fault features and the adversarial features to form an enhanced feature set; the method comprises the following steps: constructing a fault prediction model, training the model, optimizing model hyper-parameters and a network structure, collecting operation data of a storage system in real time, obtaining real-time operation features through preprocessing and feature extraction, inputting an optimal fault prediction model, and outputting a fault risk prediction value. If the predicted value exceeds a preset threshold value, triggering multi-stage fault early warning and pushing information to maintenance personnel; the fault prediction accuracy is improved through GAN data enhancement and deep learning, real-time early warning and adaptive adjustment are combined, the method is suitable for various storage systems, the operation stability of the storage systems is effectively guaranteed, and the fault loss is reduced.

Description

Technical Field

[0001] This invention belongs to the field of storage fault prediction technology, and in particular relates to a storage fault prediction method based on adversarial generation and deep learning. Background Technology

[0002] As a core component of computer systems and data centers, storage systems bear the critical functions of persistent data storage and efficient read / write operations. Their operational stability directly determines the reliability of the entire information system. Once a storage system fails, such as due to hard drive damage, I / O channel anomalies, or logical errors, it can not only disrupt critical business operations but also lead to data loss, causing irreparable economic losses and information security risks to enterprises or individuals. Therefore, predicting storage system failures in advance and intervening promptly before failures occur has become a core requirement for ensuring high availability of storage systems.

[0003] With the development of storage technology, the architecture of storage systems is becoming increasingly complex, such as distributed storage and hybrid storage arrays. Failure modes are also exhibiting multi-dimensional, non-linear, and edge-dependent characteristics. For example, hard drive failure may be triggered by a combination of factors, such as a slow increase in temperature, prolonged high read / write load, and the accumulation of occasional error logs, rather than a single factor. To address this need, the industry has developed various storage failure prediction technologies, which can be mainly divided into three categories: Traditional statistical methods rely on static statistical analysis of historical failure data, such as regression analysis and sliding window threshold judgment. These methods use manually set rules, such as identifying a hard drive temperature exceeding 55°C for 30 consecutive minutes as a precursor to a failure, to make predictions. However, these methods depend on human experience to design features and rules, and can only capture simple linear failure patterns. They have extremely low accuracy in predicting complex failures involving multiple coupled factors (such as the non-linear correlation between read / write speed degradation and error log frequency), and cannot cover rare edge failure patterns (such as occasional failures of a specific hard drive model under certain loads).

[0004] Simple machine learning methods employ algorithms such as decision trees, support vector machines, and traditional neural networks to build predictive models by training on historical data. While these methods can uncover implicit relationships in data to some extent, they are limited by model complexity and struggle to handle the high-dimensional temporal characteristics of stored data, such as multi-dimensional time-series data like continuous 24-hour temperature, read / write speed, and I / O statistics. Furthermore, they are prone to missing deep correlations in the data, resulting in poor model generalization ability.

[0005] Early deep learning methods introduced single convolutional neural networks (CNNs) or long short-term memory networks (LSTMs) for fault prediction, attempting to automatically extract features through deep learning. However, these methods have two major limitations: First, they lack effective data augmentation methods—the proportion of faulty samples in real storage systems is extremely low, and edge fault samples are scarce, making the model prone to overfitting due to insufficient training data; second, the parameters and structure of the model are fixed after training, making it unable to adapt to changes in data distribution during long-term operation of the storage system, resulting in a significant decline in prediction performance after long-term use.

[0006] In addition, existing technologies generally suffer from insufficient real-time performance and poor early warning effectiveness: some models have long training cycles and significant delays in real-time prediction, which cannot meet the real-time early warning requirements of storage systems with millisecond-level response; at the same time, early warning mechanisms are mostly triggered by single-level thresholds, which cannot provide differentiated handling suggestions based on the level of fault risk, resulting in unreasonable allocation of maintenance resources.

[0007] Therefore, existing storage fault prediction technologies have significant shortcomings in terms of prediction accuracy, feature extraction efficiency, real-time performance, and adaptability, making it difficult to meet the high reliability and high accuracy fault prediction requirements of complex storage systems. There is an urgent need for a new fault prediction method that can integrate data augmentation, automatic feature extraction, and dynamic optimization. Summary of the Invention

[0008] The purpose of this invention is to provide a storage fault prediction method based on adversarial generation and deep learning, in order to solve the technical problems that existing storage fault prediction technologies have obvious deficiencies in terms of prediction accuracy, feature extraction efficiency, real-time performance, and adaptive capability.

[0009] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A storage fault prediction method based on adversarial generation and deep learning includes the following steps: S1: Collect historical health status data and historical fault data of the storage system; preprocess the collected data to obtain preprocessed data; S2: Use a pre-set deep learning model to perform end-to-end automatic feature extraction on the pre-processed data to obtain real fault features; generate adversarial samples of simulated faults through adversarial training of a pre-set GAN, input them into the deep learning model to extract adversarial features, and fuse the real fault features with the adversarial features to obtain an enhanced feature set; S3: Construct a fault prediction model, divide the enhanced feature set into a training set, a validation set and a test set; train the model using the training set, optimize the model hyperparameters and network structure by combining the validation set with cross-validation techniques, evaluate the model performance, and continue until the model's accuracy on the test set meets the preset threshold to obtain the optimal fault prediction model. S4: Collect real-time operating data of the storage system, preprocess it and extract features to obtain real-time operating features; input the optimal fault prediction model and output the fault risk prediction value; if the fault risk prediction value exceeds the preset warning threshold, trigger multi-level fault warning and push warning information to maintenance personnel. S5: Regularly analyze the prediction results and actual fault occurrences, calculate the prediction accuracy, false alarm rate, and false negative rate; dynamically adjust the data collection frequency, the number of GAN training iterations, and the warning threshold based on the statistical results; and feed the statistical results back to the optimal fault prediction model for parameter updates or structural fine-tuning to achieve system adaptive optimization.

[0010] Preferably, the preset deep learning model is a convolutional neural network (CNN). The specific process of using the preset deep learning model to perform end-to-end automatic feature extraction on the preprocessed data in step S2 to obtain the real fault features is as follows: S21: Initial local feature capture is performed through the first convolutional layer; S22: Stable training and feature standardization are achieved through batch normalization layers; S23: Feature dimensionality reduction and key information preservation are achieved through max pooling layers; S24: Deep feature enhancement is performed through the second convolutional layer + batch normalization layer + pooling layer; S25: Feature integration and output are performed through a fully connected layer.

[0011] Preferably, in step S2, adversarial examples simulating faults are generated through iterative adversarial training between the generator and discriminator of the GAN. The adversarial examples are then input into a deep learning model to extract adversarial features, and the real fault features and adversarial features are fused according to a preset ratio. The specific process is as follows: S26: Define the loss functions for the generator and discriminator; S27: Perform iterative adversarial training of GAN to generate effective adversarial examples; S28: Generate final adversarial examples: After training stops, retain the converged generator model, input new random noise, and the generator output is the final adversarial example.

[0012] Preferably, the fault prediction model is a deep neural network (DNN), which includes an input layer, 3-5 hidden layers, and an output layer.

[0013] Preferably, the specific process of optimizing model hyperparameters and network structure by combining validation sets with cross-validation techniques in step S3, and evaluating model performance using accuracy metrics, is as follows: S31: Set the hyperparameters to be optimized and the candidate range of network structures to be tuned; S32: Hyperparameter optimization based on K-fold cross-validation; S33: Optimize the network structure based on the validation set to determine the optimal model framework.

[0014] Preferably, the specific process of step S32 is as follows: S321: Fixed network structure, traversing hyperparameter combinations; S322: Select the optimal combination of hyperparameters.

[0015] Preferably, the specific process of step S33 is as follows: S331: Use the optimal combination of hyperparameters as hyperparameters to train the model on the complete training set; S332: Structure Comparison and Selection: Traverse all candidate network structures, and calculate Acc for each structure. val In terms of overfitting, prioritize the structure with the highest accuracy.

[0016] The beneficial effects of this invention include: 1. Significantly improves fault prediction accuracy, covering complex and edge fault modes: The combination of GAN adversarial training to generate adversarial examples and enhanced feature set training model effectively solves the core problems of scarce fault samples and insufficient coverage of edge faults in existing technologies: The GAN generator, through iterative adversarial training, can generate simulated fault samples consistent with the distribution of real fault data, supplementing the missing edge fault modes in real data; at the same time, CNN, through the end-to-end feature extraction process of convolution-batch normalization-pooling, can automatically capture shallow local features and deep combined features of stored data, avoiding the limitations of manual feature design, and the DNN fault prediction model optimizes hyperparameters and network structure through K-fold cross-validation, further avoiding overfitting.

[0017] 2. Achieve efficient and automated feature extraction, reducing errors: Employing CNN as the pre-defined deep learning model, fully automated feature extraction is achieved through the following process: The first convolutional layer captures local temporal features; the batch normalization layer eliminates data distribution bias to stabilize feature extraction; the max pooling layer retains key features and reduces dimensionality; the second convolutional layer enhances deep combined features; and finally, a fully connected layer integrates and outputs true fault features. This automatically mines nonlinear correlation features from multi-dimensional stored data, not only improving feature extraction efficiency but also avoiding feature omissions or redundancy caused by manual design, significantly reducing the impact of human error on prediction results.

[0018] 3. Optimize real-time prediction and multi-level early warning to improve fault response efficiency: A low-latency processing flow is designed in the real-time prediction stage: After real-time acquisition of storage system operation data, real-time operation features are quickly generated through preprocessing logic and CNN feature extraction logic; the features are input into the optimized DNN optimal model, which can output fault risk prediction values ​​in a very short time to meet the real-time requirements of the storage system.

[0019] 4. Possesses adaptive long-term optimization capabilities, ensuring model performance stability: Through a step-by-step dynamic adjustment feedback mechanism, the model and system parameters achieve long-term adaptive optimization. Predictive metrics are periodically analyzed; if the false alarm rate is high, the warning threshold is increased; if the accuracy is low, the number of GAN training iterations is increased to optimize the quality of adversarial examples. The sampling interval is dynamically adjusted based on real-time data acquisition latency, and the statistical results are fed back to the optimal fault prediction model. Through parameter updates or structural fine-tuning, the model adapts to new data distributions. This mechanism ensures that the model maintains high predictive performance throughout the long-term operation of the storage system, eliminating the need for frequent manual retraining and significantly reducing later maintenance costs.

[0020] 5. Enhance model generalization and robustness to adapt to multi-scenario storage systems: Improve model generalization ability through two main means: First, the adversarial examples generated by GAN cover failure modes of different hardware models and different load scenarios, making the model training data more diverse; Second, K-fold cross-validation optimizes hyperparameters and network structure to avoid the model only adapting to the data distribution of a single storage system. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the storage fault prediction method based on adversarial generation and deep learning of the present invention.

[0022] Figure 2 This is a schematic diagram of the actual fault feature extraction process of the present invention. Detailed Implementation

[0023] The following is in conjunction with the appendix Figures 1-2 The present invention will be further described in detail below: Example 1 See appendix Figure 1 As shown, a storage fault prediction method based on adversarial generation and deep learning includes the following steps: S1: Collect historical health status data and historical fault data of the storage system. The data includes hard disk temperature, read / write speed, I / O operation statistics and error log data. First, perform missing value filling and outlier removal cleaning processing on the collected data. Then, use a normalization formula to standardize the cleaned data to obtain preprocessed data, so that the preprocessed data meets the input format requirements of the deep learning model.

[0024] S2: An end-to-end automatic feature extraction is performed on the preprocessed data using a preset deep learning model to obtain real fault features; at the same time, a generative adversarial network (GAN) is constructed, and adversarial samples simulating faults are generated through iterative adversarial training between the generator and discriminator of the GAN. The adversarial samples are input into the deep learning model to extract adversarial features, and the real fault features and adversarial features are fused according to a preset ratio to obtain an enhanced feature set.

[0025] S3: Construct a fault prediction model by dividing the enhanced feature set into a training set, a validation set, and a test set; train the model using the training set; optimize the model hyperparameters and network structure using the validation set and cross-validation techniques; evaluate the model performance using an accuracy metric until the model's accuracy on the test set meets a preset threshold, thus obtaining the optimal fault prediction model.

[0026] S4: Collect the operating data of the storage system in real time, and perform cleaning, normalization and feature extraction processes to obtain real-time operating features; input the real-time operating features into the optimal fault prediction model and output the fault risk prediction value; if the fault risk prediction value exceeds the preset warning threshold, trigger multi-level fault warnings and push warning information to maintenance personnel.

[0027] S5: Regularly analyze the prediction results and actual fault occurrences, calculate the prediction accuracy, false alarm rate, and false negative rate; dynamically adjust the data collection frequency, the number of training iterations of GAN, and the warning threshold of S4 based on the statistical results; and feed the statistical results back to the optimal fault prediction model for parameter updates or structural fine-tuning to achieve system adaptive optimization.

[0028] In this embodiment, the preset deep learning model is a convolutional neural network (CNN). The specific process of using a CNN to perform end-to-end automatic feature extraction on the preprocessed data in step S2 to obtain the real fault features is as follows: S21: Initial capture of local features through the first convolutional layer: Input: Temporal tensor X cnn ∈R^(1×m×d); Operation: Use k1 convolutional kernels of size (s×d), k1=16~32, s=3~5, where s is the time step window size and d is the feature dimension, ensuring coverage of all features. Calculate local feature maps using a sliding window. Formula: Conv1=ReLU(X) cnn *K1+b1); Wherein, Conv1 is the original output feature of the first convolutional layer, K1 is the weight matrix of the first convolutional layer, b1 is the bias, and ReLU is the activation function. Nonlinearity is introduced through the ReLU activation function to simulate the nonlinear correlation of fault features. Output: The first convolutional feature Conv1∈R^(1×(m-s+1)×k1), where the dimension means batch × effective time step × number of convolutional kernels. Each convolutional kernel corresponds to one local feature, such as the positive correlation between temperature and reading speed. S22: Stable training and feature standardization are achieved through batch normalization layers; Operation: Batch normalize each dimension of Conv1 convolutional kernels to eliminate the impact of data distribution offset on feature extraction. Batch normalization formula: BN1=(Conv1-μ1) / (σ1²+ε) 1 / 2 Where μ1 is the mean of Conv1, σ1 is the standard deviation, and ε = 1e -5 To prevent tiny values ​​with a denominator of 0; The above process can avoid the model from focusing too much on a feature due to excessive fluctuations in the value of a certain feature, and ensure that all fault-related features are extracted in a balanced manner.

[0029] S23: Feature dimensionality reduction and key information preservation are achieved through max pooling layers; Operation: Use a pooling window of size (p×1), p=2, downsample only the time step dimension, preserve features in the convolution kernel dimension, and take the maximum value within the window for BN1: Formula: Pool1=MaxPool(BN1, pool size =(p,1)); Output: Pooled features Pool1∈R^(1×((m-s+1) / p)×k1), which reduce the time step dimension by downsampling and retain the strongest fault feature signal of each local window, such as the maximum temperature deviation value within a certain window.

[0030] S24: Deep feature enhancement is performed through the second convolutional layer + batch normalization layer + pooling layer; Repeat steps S21-S23, using k2=32~64 convolutional kernels of size (s'×k1) (s'=2~3) to further extract combined features of local features, such as the association pattern of temperature features + error log features in Pool1. Finally, the second pooling feature Pool2∈R^(1×t×k2) is output, where t is the final time step dimension, t< <m。

[0031] S25: Feature integration and output are performed through a fully connected layer; Operation: Flatten Pool2 into a one-dimensional vector Flatten∈R^(t×k2), input it into a fully connected layer (number of neurons n=128~256), and integrate all deep local features through linear transformation: Formula: F cnn =ReLU(Flatten*W fc +b fc ), where W fc b represents the weights of the fully connected layer. fc For bias; Output: F cnn∈R^n, which is the real fault feature vector extracted based on CNN. Each element in the vector corresponds to a deep feature that is strongly correlated with the fault, such as the composite feature of temperature fluctuation amplitude × error log frequency.

[0032] Example 2 Based on Example 1, in step S2, adversarial examples simulating faults are generated through iterative adversarial training of the generator and discriminator of the GAN. The adversarial examples are then input into a deep learning model to extract adversarial features, and the real fault features and adversarial features are fused according to a preset ratio. The specific process is as follows: S26: Define the loss functions for the generator and discriminator: Discriminator Loss (LossD): The goal is to distinguish between real and simulated data as accurately as possible. The formula is: LossD=-E[log(D(X real ))]-E[log(1-D(X fake ))]; Where: X real The actual preprocessed data output in step S1 is labeled 1, representing the actual data. X fake : Simulated fault data output by the generator, labeled 0, indicating generation; D(·): The output probability of the discriminator; E[·]: Take the expected value of all samples in the batch to ensure that the loss reflects the overall performance of the batch; Generator Loss (LossG): The goal is to make the generated simulated data deceive the discriminator as much as possible, so that D(X) fake )≈1), the formula is: LossG=-E[log(D(X fake ))]; If the generator has good performance, D(X) fake ) is close to 1, log(D(X) fake When the sum of all possible values ​​is close to 0, the loss value (LossG) approaches 0. S27: Perform iterative adversarial training of GAN to generate effective adversarial examples: By alternating between fixing one side and optimizing the other, the generator and discriminator gradually reach Nash equilibrium, meaning that the discriminator cannot distinguish between real and generated data, and the accuracy is close to 50%. In the end, high-quality adversarial examples are generated to simulate real failure modes and supplement the missing edge failure samples in the real data. S271: Fixed generator, optimized discriminator, improved discrimination capability; B samples are randomly selected from the real preprocessed data pool in step S1, where B = 32~64, and denoted as X. real_batch ; Generate B random noise Z batch (Dimension Z) dim Inputting data into the generator yields B simulated fault data points X. fake_batch =Generator(Z batch ); X real_batch (Label 1) and X fake_batch (Label 0) Mixed input discriminator, calculate discriminator output D(X) real_batch ) and D(X fake_batch) ; The discriminator loss for the current batch is calculated based on the LossD formula. The network parameters of the discriminator are updated by gradient descent (only the discriminator weights are updated, while the generator parameters remain fixed) until the LossD decreases and tends to stabilize.

[0033] Fixed discriminator, optimized generator, improved deception capability; Generate B new random noise Z new_batch Input the data into the generator to obtain new simulated fault data X. fake_new_batch ; X fake_new_batch A discriminator with fixed input parameters yields a discriminant output D(X). fake_new_batch ); The generator loss for the current batch is calculated based on the LossG formula. The network parameters of the generator are updated by gradient descent (only the generator weights are updated, while the discriminator parameters remain fixed) until LossG decreases and tends to stabilize. Training stopping conditions to ensure the effectiveness of adversarial examples: Repeat the iterative process of optimizing the discriminator → optimizing the generator until the following two conditions are met: Discriminator for mixed batches (X) real +X fake The discrimination accuracy of the data is stable between 45% and 55%, indicating that the discriminator can no longer effectively distinguish between real and generated data, and the samples generated by the generator have a similar failure mode to the real data. Both the generator loss Loss_G and the discriminator loss LossD tend to converge (the fluctuation range of the loss value is <5% in 10 consecutive iterations), avoiding training oscillations and ensuring stable generator output.

[0034] S28: Generate the final adversarial example After training stops, retain the converged generator model and input a new random noise Z. final (Consistent with the noise distribution during the training phase, dimension Z) dim The generator output is the final adversarial example, denoted as X. adv .

[0035] Key verification: Randomly select 100 X records adv If the discrimination accuracy is still between 45% and 55%, then confirm X. adv These can be used as effective adversarial examples for subsequent feature extraction.

[0036] Example 3 Based on Example 1 or Example 2, the fault prediction model is a deep neural network (DNN). The DNN includes an input layer (with the same dimension as the enhanced feature set), 3-5 hidden layers (each layer has 64-256 neurons, and the activation function is ReLU), and an output layer (the activation function is Sigmoid, and the output is a single-dimensional fault risk value).

[0037] The specific process of optimizing model hyperparameters and network structure using validation set and cross-validation techniques in step S3, and evaluating model performance using accuracy metrics, is as follows: The candidate ranges for setting the hyperparameters to be optimized and the network structures to be tuned are shown in Table 1 below: Table 1. Candidate range of hyperparameters to be optimized and network structures to be tuned. Network architecture candidate range: DNN architecture candidates, with core optimization focusing on the number of hidden layers and the number of neurons per layer: Candidate structures can be: Candidate 1: Input layer (dimension = augmented feature dimension) → 2 hidden layers (64 → 32) → Output layer (1-dimensional, Sigmoid); Candidate 2: Input layer → 3 hidden layers (128 → 64 → 32) → Output layer; Candidate 3: Input layer → 3 hidden layers (256 → 128 → 64) → Output layer; S31: Hyperparameter optimization based on K-fold cross-validation: Using 5-fold cross-validation on the training set D train With validation set D val Optimize hyperparameters on the merge set, merge set D train_val =D train ∪D val .

[0038] S32: Network structure tuning based on the validation set to determine the optimal model framework: using the optimal hyperparameter set HP opt For fixed conditions, in the validation set D val By comparing the performance of different network structures, the optimal network structure (Struct) can be determined. opt Specific steps: The specific process of step S31 is as follows: S311: Fixed network structure, traversing hyperparameter combinations; Generate hyperparameter combinations, such as lr=1e-3, batch_size=64, epoch=100, optimizer=Adam, activation function=ReLU; Perform 50% cross-validation: Round 1: Train the model on temporary training sets S1~S4 → Calculate the accuracy Acc1 on temporary validation set S5; Round 2: Training on temporary training sets S1~S3, S5 → Calculating Acc2 on temporary validation set S4; Repeat this process 5 times to obtain Acc1~Acc5; Calculate the average accuracy Acc of this hyperparameter combination. avg =(Acc1+Acc2+Acc3+Acc4+Acc5) / 5 (Accuracy formula is the same as above: Accuracy=Correct) Predictions / Total Predictions ).

[0039] S312: Select the optimal combination of hyperparameters: Iterate through all hyperparameter combinations (e.g., 10 learning rates × 3 batch sizes × 4 epochs = 120 combinations), and calculate Acc for each combination. avg ; Select Acc avg The highest combination of hyperparameters is denoted as the optimal hyperparameter set HP. opt , Such as HP opt ={lr=5e -4 batch size =64, epoch=150, optimizer=Adam, activation function=ReLU}.

[0040] The specific process of step S32 is as follows: S321: Using HP opt As a hyperparameter, in the complete training set D train Train the model to avoid subset splitting for cross-validation and make full use of the training data; Input the trained model into the validation set D val Calculate the verification accuracy Acc val (Formula as above), and simultaneously record the degree of overfitting. If Acc train (Training set accuracy) - Acc val If the percentage is greater than 10%, it is considered overfitting.

[0041] S322: Structural Comparison and Screening: Iterate through all candidate network architectures, and compute Acc for each architecture. val And the degree of overfitting; Filtering rules: Prioritize Acc val The highest structure; If multiple structures Acc val If the values ​​are the same or the difference is less than 2%, select the structure with the lowest degree of overfitting.

[0042] In step S4, key operational data is collected from the hardware modules and software logs of the storage system at the millisecond level to ensure that the data covers the core monitoring dimensions of fault precursors and is consistent with the parameter dimensions of historical data collected in S1, thus ensuring compatibility with subsequent preprocessing and feature extraction.

[0043] The preprocessing process includes real-time data cleaning to eliminate noise and anomalies: To address potential issues in real-time data such as sensor fluctuations, packet loss, and false log reports, the following cleaning operations are performed: Missing value imputation: If the parameter value of a certain time step k is missing, the sliding window mean filling method is used to fill it with the mean of the previous 5 valid time steps to avoid the missing value of a single time step affecting the overall features. Outlier Removal and Correction: Outliers are identified using the 3σ principle: First, the mean μ of each parameter is calculated based on historical data. hist With standard deviation σ hist ; If the real-time parameter value satisfies |X real_k -μ hist |>3σ hist It is determined to be an outlier, using μ hist +3σ hist (Upper limit) or μ hist -3σ hist (Lower limit) correction to avoid extreme values ​​interfering with feature extraction.

[0044] Real-time data normalization to unify numerical scale: Mapping the cleaned real-time data to the [0,1] interval: Normalization formula: x real =(X real_clean -μ hist ) / (max hist -min hist ); Where X real_clean The real-time parameter values ​​after cleaning, μ hist ,max hist min hist All values ​​are statistical values ​​from historical data, not real-time data, to avoid deviations in normalization results caused by fluctuations in the range of real-time data. The final output is the preprocessed real-time data X. real_pre .

[0045] Real-time feature extraction: Fully reuses the pre-defined deep learning model for S2 real fault feature extraction, and applies it to X. real_pre Transformed into real-time running feature F real_time This ensures that its feature dimensions and semantics are consistent with the S3 enhanced feature set, and can be directly input into the optimal fault prediction model.

[0046] CNN Real-time Feature Extraction: Reuse the CNN model trained in S2 and perform the following feature extraction steps (corresponding to S21-S25 in S2): Failure risk prediction: F real_time Input the optimal fault prediction model trained by S3, and output the fault risk prediction value Y through forward calculation of the model. real_time The core is to ensure the real-time nature of model calls and the interpretability of the output.

[0047] Forward computation process: Input layer: Receives F real_time (128 dimensions), matching the dimensions of the DNN input layer; Hidden layers: feature dimensions are gradually compressed through 3-5 layers of fully connected layers activated by ReLU. Output layer: The output is mapped to the [0,1] interval using the Sigmoid activation function to obtain the fault risk prediction value Y. real_time , such as Y real_time =0.75 indicates that the current storage system has a 75% risk of failure.

[0048] Prediction result verification: To avoid invalid predictions caused by model anomalies, a result reasonableness verification is added: If Y real_time If the value exceeds the [0,1] interval, it is determined to be a prediction anomaly, and the model is automatically reloaded and prediction is performed again. If the prediction results are abnormal for three consecutive times, a model fault warning will be triggered, which is independent of the storage fault warning, prompting maintenance personnel to check the model deployment status.

[0049] Multi-level fault warning triggering and information push: According to Y real_time The system compares the warning threshold with the preset warning threshold, triggers the corresponding level of warning, and pushes the warning information through multiple channels to ensure that maintenance personnel respond in a timely manner.

[0050] Preset warning thresholds and level classifications: Based on the S3 test set verification results and storage system reliability requirements, three levels of early warning thresholds and corresponding response strategies are preset, as detailed in Table 2 below: Table 2 Warning Status and Response Strategies

Claims

1. A storage fault prediction method based on adversarial generation and deep learning, characterized in that, Includes the following steps: S1: Collect historical health status data and historical fault data of the storage system; preprocess the collected data to obtain preprocessed data; S2: Use a pre-set deep learning model to perform end-to-end automatic feature extraction on the pre-processed data to obtain real fault features; Adversarial samples simulating faults are generated through adversarial training of a pre-defined GAN, input into a deep learning model to extract adversarial features, and then the real fault features are fused with the adversarial features to obtain an enhanced feature set. S3: Construct a fault prediction model and divide the enhanced feature set into a training set, a validation set, and a test set; The model is trained using the training set, and the model hyperparameters and network structure are optimized by combining the validation set with cross-validation techniques. The model performance is evaluated until the accuracy of the model on the test set meets the preset threshold, thus obtaining the optimal fault prediction model. S4: Real-time acquisition of operating data from the storage system, preprocessing and feature extraction to obtain real-time operating features; input of the optimal fault prediction model, output of fault risk prediction value; If the predicted fault risk exceeds the preset warning threshold, a multi-level fault warning will be triggered and warning information will be pushed to maintenance personnel. S5: Regularly analyze the prediction results and actual failure occurrences, and calculate the prediction accuracy, false alarm rate and false negative rate; The data collection frequency, the number of training iterations of the GAN, and the early warning threshold are dynamically adjusted based on the statistical results. At the same time, the statistical results are fed back to the optimal fault prediction model for parameter updates or structural fine-tuning to achieve adaptive optimization of the system.

2. The storage fault prediction method based on adversarial generation and deep learning according to claim 1, characterized in that, The preset deep learning model is a convolutional neural network (CNN). The specific process of using the preset deep learning model to perform end-to-end automatic feature extraction on the preprocessed data to obtain the real fault features in step S2 is as follows: S21: Initial local feature capture is performed through the first convolutional layer; S22: Stable training and feature standardization are achieved through batch normalization layers; S23: Feature dimensionality reduction and key information preservation are achieved through max pooling layers; S24: Deep feature enhancement is performed through the second convolutional layer + batch normalization layer + pooling layer; S25: Feature integration and output are performed through a fully connected layer.

3. The storage fault prediction method based on adversarial generation and deep learning according to claim 2, characterized in that, In step S2, adversarial examples simulating faults are generated through iterative adversarial training between the generator and discriminator of the GAN. The adversarial examples are then input into the deep learning model to extract adversarial features, and the real fault features and adversarial features are fused according to a preset ratio. The specific process is as follows: S26: Define the loss functions for the generator and discriminator; S27: Perform iterative adversarial training of GAN to generate effective adversarial examples; S28: Generate final adversarial examples: After training stops, retain the converged generator model, input new random noise, and the generator output is the final adversarial example.

4. The storage fault prediction method based on adversarial generation and deep learning according to claim 1, characterized in that, The fault prediction model is a deep neural network (DNN), which includes an input layer, 3-5 hidden layers, and an output layer.

5. A storage fault prediction method based on adversarial generation and deep learning according to claim 4, characterized in that, The specific process of optimizing model hyperparameters and network structure using validation set and cross-validation techniques in step S3, and evaluating model performance using accuracy metrics, is as follows: S31: Set the hyperparameters to be optimized and the candidate range of network structures to be tuned; S32: Hyperparameter optimization based on K-fold cross-validation; S33: Optimize the network structure based on the validation set to determine the optimal model framework.

6. The storage fault prediction method based on adversarial generation and deep learning according to claim 5, characterized in that, The specific process of step S32 is as follows: S321: Fixed network structure, traversing hyperparameter combinations; S322: Select the optimal combination of hyperparameters.

7. The storage fault prediction method based on adversarial generation and deep learning according to claim 5, characterized in that, The specific process of step S33 is as follows: S331: Use the optimal combination of hyperparameters as hyperparameters to train the model on the complete training set; S332: Structure Comparison and Selection: Traverse all candidate network structures, and calculate Acc for each structure. val In terms of overfitting, prioritize the structure with the highest accuracy.

Images