Disk fault processing method and device, equipment and medium

Abstract

The invention discloses a disk fault processing method and device, equipment and a medium, and relates to the technical field of storage, and the method comprises the following steps: obtaining running state data of each disk, and obtaining a system load and system disk redundancy; performing line feature extraction on the running state data of each disk; performing first filtering on all the disk operation state data after the line feature extraction through a first model to obtain a first risk disk set; filtering the first risk disk set for the second time through a second model to obtain a second risk disk set; performing risk grade division on the second risk disk set through a third model, the system load and the system disk redundancy, and determining a target control strategy corresponding to the disk according to the disk risk grade; and performing fault processing on the disk according to the target control strategy corresponding to the disk. According to the technology provided by the invention, multi-level anomaly detection can be effectively carried out on the disk fault, so that the accuracy and reliability of disk fault processing are improved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a disk failure processing method, apparatus, device and medium. Background Art

[0002] Currently, disk failures are the main cause of data loss and service interruptions in server storage systems. The biggest technical pain points of disk failure handling technologies are: the inability to prevent failures through passive response processing; the lack of effective control over the risk of failure propagation; insufficient intelligence in fault handling, and low operation and maintenance efficiency. Summary of the Invention

[0003] The present application provides a disk failure processing method, apparatus, device and medium, the method comprising obtaining the operating status data of each disk, obtaining the system load, and the system disk redundancy; performing line feature extraction on the operating status data of each disk, the line features including time domain features, frequency domain features and topological features; performing a first filtering on all disk operating status data after line feature extraction using a first model to obtain a first risk disk set; performing a second filtering on the first risk disk set using a second model to obtain a second risk disk set; dividing the second risk disk set into risk levels using a third model, the system load and the system disk redundancy, and determining the target control strategy corresponding to the disk according to the disk risk level; and performing disk failure processing according to the target control strategy corresponding to the disk. The technology of the present application can effectively perform multi-level anomaly detection on disk failures, thereby improving the accuracy and reliability of disk failure processing.

[0004] This application provides a disk failure processing method, which is applied to a server storage system. The system includes several disks. The method includes:

[0005] Obtain the operating status data of each disk; obtain the system load and system disk redundancy;

[0006] Extract line features from the operating status data of each disk. Line features include time domain features, frequency domain features, and topological features.

[0007] Performing a first filtering on the disk operation status data after all line features are extracted using the first model to obtain a first risk disk set;

[0008] Performing a second filtering on the first risk disk set using the second model to obtain a second risk disk set;

[0009] The risk level of the second risk disk set is divided according to the third model, the system load and the system disk redundancy, and the target control strategy corresponding to the disk is determined according to the disk risk level;

[0010] Handle disk failures according to the target control policy corresponding to the disk.

[0011] The present application also provides a disk failure processing device, comprising:

[0012] The acquisition module is used to obtain the operating status data of each disk; obtain the system load and system disk redundancy;

[0013] An extraction module is used to extract line features from the operating status data of each disk. The line features include time domain features, frequency domain features, and topological features.

[0014] A first filtering module is configured to perform a first filtering on the disk operation status data after all line features are extracted using a first model to obtain a first risk disk set;

[0015] a second filtering module, configured to perform a second filtering on the first risk disk set using a second model to obtain a second risk disk set;

[0016] a determination module, configured to classify the risk level of the second risk disk set by using the third model, the system load, and the system disk redundancy, and determine the target control strategy corresponding to the disk according to the disk risk level;

[0017] The processing module is used to process disk failures according to a target control strategy corresponding to the disk.

[0018] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of a disk failure processing method when executing the computer program. The method comprises:

[0019] Obtain the operating status data of each disk; obtain the system load and system disk redundancy;

[0020] Extract line features from the operating status data of each disk. Line features include time domain features, frequency domain features, and topological features.

[0021] Performing a first filtering on the disk operation status data after all line features are extracted using the first model to obtain a first risk disk set;

[0022] Performing a second filtering on the first risk disk set using the second model to obtain a second risk disk set;

[0023] The risk level of the second risk disk set is divided according to the third model, the system load and the system disk redundancy, and the target control strategy corresponding to the disk is determined according to the disk risk level;

[0024] Handle disk failures according to the target control policy corresponding to the disk.

[0025] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of a disk failure processing method are implemented, and the method includes:

[0026] Obtain the operating status data of each disk; obtain the system load and system disk redundancy;

[0027] Extract line features from the operating status data of each disk. Line features include time domain features, frequency domain features, and topological features.

[0028] Performing a first filtering on the disk operation status data after all line features are extracted using the first model to obtain a first risk disk set;

[0029] Performing a second filtering on the first risk disk set using the second model to obtain a second risk disk set;

[0030] The risk level of the second risk disk set is divided according to the third model, the system load and the system disk redundancy, and the target control strategy corresponding to the disk is determined according to the disk risk level;

[0031] Handle disk failures according to the target control policy corresponding to the disk.

[0032] The present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the disk failure processing method are implemented. The method includes:

[0033] Obtain the operating status data of each disk; obtain the system load and system disk redundancy;

[0034] Extract line features from the operating status data of each disk. Line features include time domain features, frequency domain features, and topological features.

[0035] Performing a first filtering on the disk operation status data after all line features are extracted using the first model to obtain a first risk disk set;

[0036] Performing a second filtering on the first risk disk set using the second model to obtain a second risk disk set;

[0037] The risk level of the second risk disk set is divided according to the third model, the system load and the system disk redundancy, and the target control strategy corresponding to the disk is determined according to the disk risk level;

[0038] Handle disk failures according to the target control policy corresponding to the disk.

[0039] Through the present application, the method includes obtaining the operating status data of each disk; obtaining the system load and the system disk redundancy; extracting line features from the operating status data of each disk, wherein the line features include time domain features, frequency domain features, and topological features; performing a first filtering on the disk operating status data after all line features are extracted through a first model to obtain a first risk disk set; performing a second filtering on the first risk disk set through a second model to obtain a second risk disk set; dividing the second risk disk set into risk levels through a third model, the system load, and the system disk redundancy, and determining the target control strategy corresponding to the disk according to the disk risk level; and performing disk fault processing according to the target control strategy corresponding to the disk. The technology of the present application can effectively perform multi-level anomaly detection on disk faults, thereby improving the accuracy and reliability of disk fault processing.

[0040] The technical solution of this application 1. effectively improves the disk failure prediction performance: the fault detection rate reaches 92.4% (the related technology is 63.1%); the false alarm rate is reduced to 5.2% (the related technology is 28.7%); the average warning advance time reaches 37 hours; 2. optimizes the disk failure operation and maintenance efficiency: the fault handling time is shortened by 68%; the unexpected downtime is reduced by 83%; the hard disk replacement cost is reduced by 42%; 3. The system reliability is enhanced: the disk redundant array reconstruction failure rate is reduced from 15% to 2%; the data loss incidents are reduced by 91%; the storage system availability reaches 99.9%. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 A first flow chart of the disk failure handling method provided in an embodiment of the present application;

[0043] Figure 2 A flow chart of disk filtering control provided in an embodiment of the present application;

[0044] Figure 3 A specific flow chart of the disk failure handling method provided in an embodiment of the present application;

[0045] Figure 4 A first structural diagram of a disk failure handling device provided in an embodiment of the present application;

[0046] Figure 5 The exemplary systems provided for the embodiments of the present application can be used to implement the various embodiments described in the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0048] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0049] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0050] Related disk failure handling solutions mainly rely on the following technologies:

[0051] (1) SMART (Self-Monitoring, Analysis and Reporting Technology) hard disk self-monitoring technology: monitors the health status of the hard disk through preset thresholds, but has the following shortcomings:

[0052] It can only detect obvious faults but cannot predict potential faults;

[0053] Static thresholds cannot adapt to different workloads and environmental changes;

[0054] High false positive rate, leading to unnecessary hard drive replacement;

[0055] (2) RAID (Redundant Array of Independent Disks) disk redundancy mechanism: Data reconstruction after a failure occurs, but it has obvious defects:

[0056] The rebuild process will significantly increase the load on other hard drives;

[0057] The reconstruction time window may trigger a cascading failure;

[0058] There is no way to prevent disk failures;

[0059] (3) Machine learning method: Using static models to analyze SMART data, but there are the following problems:

[0060] The model update cycle is long and cannot adapt to new failure modes in real time;

[0061] Feature engineering relies on manual experience and has limited generalization capabilities;

[0062] Lack of comprehensive consideration of the overall status of the system.

[0063] The embodiment of the present application provides a method for handling disk failures, such as Figure 1 As shown, the method is applied to a server storage system, the system includes several disks, and the method includes:

[0064] Obtain the operating status data of each disk; obtain the system load and system disk redundancy;

[0065] Extract line features from the operating status data of each disk. Line features include time domain features, frequency domain features, and topological features.

[0066] Performing a first filtering on the disk operation status data after all line features are extracted using the first model to obtain a first risk disk set;

[0067] Performing a second filtering on the first risk disk set using the second model to obtain a second risk disk set;

[0068] The risk level of the second risk disk set is divided according to the third model, the system load and the system disk redundancy, and the target control strategy corresponding to the disk is determined according to the disk risk level;

[0069] Handle disk failures according to the target control policy corresponding to the disk.

[0070] It is understandable that this application proposes a server hard disk failure prediction and isolation method based on deep learning, (1) dynamic learning architecture:

[0071] Long Short-Term Memory (LSTM) is used to process time series SMART data.

[0072] Introducing online learning mechanism to update model parameters in real time;

[0073] Combine reinforcement learning to optimize fault judgment strategies;

[0074] (2) Multi-dimensional monitoring and collection:

[0075] Hardware level: SMART raw data (Reallocated Sector Count, Spin Up Time, etc.);

[0076] System level: hard disk I / O latency, read and write throughput;

[0077] Environmental level: temperature, vibration frequency;

[0078] (3) Hierarchical response mechanism:

[0079] Warning level (risk probability 30-60%): Slow down the system and perform data cold migration.

[0080] High-risk level (risk probability 60-85%): Perform hot migration + restrict write;

[0081] Emergency level (risk probability > 85%): Isolate the hard drive and start rebuilding.

[0082] The embodiment of the present application provides a method for handling disk failures, such as Figure 1 As shown, the method is applied to a server storage system, the system includes several disks, and the method includes:

[0083] The technical solution of this application adopts a modular layered design and includes the following core components: intelligent perception layer, streaming processing layer, and hybrid model layer.

[0084] Step S01, obtaining the operating status data of each disk; obtaining the system load and system disk redundancy;

[0085] RAID (Redundant Array of Independent Disks) is a technology that combines multiple physical disk drives into one or more logical units to improve data redundancy, performance, or both. Different RAID levels provide different levels of redundancy and performance advantages.

[0086] Choosing the right RAID level depends on factors such as performance, cost, and reliability. For applications that require high reliability and fault tolerance, RAID 1, RAID 5, RAID 6, or RAID 10 are usually chosen. For scenarios that prioritize performance over data security, RAID 0 may be considered.

[0087] Step S011, collecting the operating status data of each disk;

[0088] The disk's operating status data includes disk health parameter data (SMART parameter data), disk performance data, and system environment data;

[0089] Disk performance data includes disk input / output latency, read / write throughput, and load rate;

[0090] System environment data includes ambient temperature and vibration frequency;

[0091] The operating status data of each disk is stored in the message queue.

[0092] Here, the intelligent perception layer: The intelligent perception layer is the cornerstone of data collection in this application. Its design achieves a precise balance between resource usage, data accuracy, and security. The following is an explanation from the perspective of technical implementation:

[0093] 1. Micro-agent design: (1) Minimalist architecture design: The Rust language is used for memory control, and the memory usage is less than 50MB through static memory pre-allocation (to avoid dynamic allocation fragmentation), zero-copy data processing (such as the bytes library directly operates on raw bytes), and polling instead of interrupts (to reduce context switching overhead).

[0094] (2) Modular functional components: A dynamic loading mechanism is used to load acquisition modules on demand (such as NVMe log collectors dedicated to NVMe disks) and a plug-in architecture is used to support hot-swappable sensor drivers (temperature / vibration / current, etc.).

[0095] 2. High-frequency data collection: (1) Full SMART parameter extension: not only collects standard parameters (such as Reallocated_Sectors), but also includes manufacturer-specific items; (2) Collects information such as hard disk I / O latency, read and write throughput, and load rate; (3) Dynamic frequency adjustment strategy: Dynamic frequency adjustment based on hard disk load rate and working hours; Sudden abnormalities (such as a sudden temperature rise of 5°C) immediately trigger additional data collection event triggering mode.

[0096] The collected data is pre-processed lightly: (1) Sliding window statistics: using the Welford algorithm to calculate the standard deviation in real time and memory-efficiently; (2) A three-level filtering mechanism for anomaly detection: as shown in Table 1 below:

[0097] Table 1

[0098]

[0099] Secure transmission guarantee: Data integrity check: A double-layer protection mechanism based on frame-level CRC32 (additional checksum to each 512-byte data packet) and streaming HMAC (generating a signature (SHA3-256) every 60 seconds).

[0100] Step S02 : extracting line features from the operating status data of each disk. The line features include time domain features, frequency domain features, and topological features.

[0101] Specifically, the stream processing (Flink real-time processing engine) sub-steps:

[0102] (1) Read data stream from Kafka message queue;

[0103] Flink subscribes to and consumes data streams from Kafka message queues, implementing end-to-end real-time data pipelines.

[0104] (2) Feature calculation

[0105] Online feature extraction is performed on each piece of disk data, mainly including the following three types of features:

[0106] A. Time Domain Characteristics

[0107] Mean: reflects the overall level of the signal;

[0108] Kurtosis: measures the thickness of the distribution tail and identifies abnormal fluctuations;

[0109] Hurst Index: Determines the long-term correlation of time series through R / S analysis and is used to predict trend persistence or anti-persistence.

[0110] B. Frequency Domain Characteristics

[0111] Wavelet energy spectrum:

[0112] Discrete wavelet transform (DWT) was performed using Daubechies-4 wavelet;

[0113] Sum the squares of the detail coefficients of each layer to get the energy of the frequency band;

[0114] Can be used to identify high-frequency transient events or abnormal signals.

[0115] C. Topological features

[0116] Rack Dependency:

[0117] Analyze the correlation between the operating status of different disks in the same rack;

[0118] If a hard drive (disk) behaves significantly differently from the others, it may indicate a potential failure risk.

[0119] Step S021, calculating the time domain characteristics of the disk operation status data by a sliding window method;

[0120] Use the Welford online algorithm to calculate the mean and variance of disk operation status data;

[0121] The kurtosis of disk operation status data is calculated by the fourth-order central moment;

[0122] The long-range correlation of disk operation status data was evaluated by rescaled range analysis;

[0123] The frequency domain characteristics of the disk's operating status data are extracted by wavelet transform, and orthogonal wavelet is set as the mother wavelet;

[0124] Calculating the energy spectrum of the disk operation status data according to the sum of squares of detail coefficients of the disk operation status data;

[0125] The topological characteristics of the disk's operating status data are calculated using the continuous coherence method, and the operating status correlation between different disks in the same rack is analyzed.

[0126] Specifically, the streaming layer is the core engine for implementing real-time fault prediction in this application. The distributed data processing pipeline built on Apache Flink achieves millisecond-level conversion from raw data to high-level features. The following is a complete technical analysis from architecture design to feature calculation:

[0127] 1. Real-time processing pipeline architecture: (1) Data ingestion layer: Implement the CheckpointedFunction function interface through a custom KafkaSource queue to ensure Exactly-Once semantics and hash distribution by diskId, ensuring that the same hard disk data is always processed by the same Task partition to achieve Kafka source adaptation optimization; (2) Window processing: Based on the tumbling window (TumblingEventTimeWindows) fixed 5-minute window type, BoundedOutOfOrdernessTimestampExtractor allows 10 seconds of disorder watermark strategy and RocksDBStateBackend (disk overflow to prevent OOM) state backend key parameter configuration.

[0128] 2. Feature engineering implementation: (1) Time domain feature calculation: Sliding statistic optimization is achieved through three indicators: mean / variance: using the Welford online algorithm (numerically stable), kurtosis: calculated based on the fourth-order central moment (reflecting the sharpness of the distribution), and Hurst index: evaluating long-range correlation through rescaled range analysis (R / S); (2) Frequency domain feature extraction: wavelet transform is achieved by using the WaveletTransform class of Apache Commons Math, mother wavelet selection: Daubechies-4 (suitable for transient feature detection), and energy spectrum calculation: summing the squares of detail coefficients.

[0129] 3. Performance optimization technologies: (1) Computational acceleration: NI native extension: Call C++ to implement complex operations such as Hurst exponent calculation (8 times faster); Vectorized computing: Use Intel MKL library to accelerate matrix operations; (2) State management: Use Delta encoding to compress time series data and use Zstd to compress state snapshots (compression ratio 3:1) to achieve window state compression.

[0130] 4. Fault tolerance and monitoring: (1) Checkpoint mechanism: A checkpoint is generated every 30 seconds, and the timeout threshold is set to 10 minutes; (2) Metric monitoring system: Key metrics are exposed through three parameters: numRecordsInPerSecond: input throughput, currentOutputWatermark: event time progress, and stateSize: operator state size; (3) Backpressure handling: Short-term backpressure is adopted: increasing the window sliding step size; continuous backpressure: a dynamic backpressure strategy that degrades feature calculation accuracy (such as disabling the Hurst index).

[0131] At the same time, the dynamic weights are adjusted: the feature importance is adjusted according to the hard disk usage time (for example, aging hard disks focus on bad sector counts, and new hard disks focus on temperature fluctuations).

[0132] Automatically adjust feature weights based on drive usage: For aging drives, focus on Reallocated SectorCount; for new drives, focus on Temperature Variance.

[0133] New hard drives (<1 year): The core risks are defects and environmental adaptability, so the focus is on hard drive temperature fluctuations; aged hard drives (>3 years): The core risks are mechanical wear and media degradation, so the focus is on bad sector counts and seek error rates; for hard drives between new and aged, the feature weights of new and aged hard drives need to be balanced.

[0134] Step S03, as Figure 2 As shown, all disk operation status data after line feature extraction are filtered for the first time using the first model to obtain a first risk disk set.

[0135] Specifically, the hybrid model layer is the core module for this application to achieve intelligent fault prediction and decision-making. Through a three-level progressive model architecture, it achieves the optimal balance between speed and accuracy. The following is a complete technical analysis from model design to collaborative mechanism:

[0136] The design principle of the three-level model architecture: (1) Hierarchical division of labor strategy: as shown in Table 2 below:

[0137] Table 2

[0138]

[0139] Step S031, obtain the gradient boosting decision tree prediction function of the first model

[0140]

[0141] Where K is the total number of gradient boosting decision trees, f kis the prediction function of the k-th tree, For all possible decision tree spaces, x i is the input line feature vector data of the i-th disk;

[0142] By formula: Calculate the first risk probability value (initial risk probability value) P for each disk LightGBM , σ is the activation function;

[0143] Determine whether the initial risk probability value of any disk is less than a first preset value (30%);

[0144] If yes, the disk is a normal disk and the process ends; if no, the disk is classified into the first risk disk set;

[0145] Among them, the first model is the gradient boosting decision tree model LightGBM (Light Gradient Boosting Machine).

[0146] Specifically, here is the objective function:

[0147] Binary cross entropy loss function The difference between the predicted probability distribution and the true label;

[0148] y i : true label;

[0149] Sigmoid function;

[0150] l: loss value of a single sample;

[0151] Ω(f k ): Regularization term

[0152] T: number of leaf nodes (controls tree complexity);

[0153] ω: leaf weight;

[0154] λ: regularization coefficient;

[0155] Ω: overall optimization goal (objective function value);

[0156] γ: time penalty coefficient;

[0157] By adding a regularization term to the loss function, we can predict whether the disk will fail.

[0158] Using gain split contribution:

[0159]

[0160] G t : The sum of the first-order gradient of the left subtree of the split point t

[0161] H t : The sum of the second-order gradients of the left subtree of the split point t

[0162] v t : Split feature index

[0163] When the variable v t = squared measure of “contribution strength” or “importance” when j = j;

[0164] T: total number of time steps (e.g. number of SMART data collection points);

[0165] t: time index;

[0166] By calculating the different states It can be judged that:

[0167] In which states (such as "warning level") do certain SMART attributes experience abnormal fluctuations?

[0168] And whether specific intervention measures are needed for this state (such as slowing down operation, migrating data, etc.).

[0169] Step S04, as Figure 2 As shown, the first risk disk set is filtered for the second time using the second model to obtain the second risk disk set.

[0170] Step S041: Obtain the input line feature vector data x of the i-th disk in the first risk disk set. i , the forward weight matrix W → , the forward hidden state vector of the second model at the previous time step t

[0171] By formula: Calculate the second model forward hidden state vector

[0172] Get the inverse weight matrix W ← , the reverse hidden state vector of the second model at the next time step t

[0173] By formula: Calculate the second model reverse hidden state vector

[0174] By formula: Calculate the second model hidden state vector h t ;

[0175] Get the transformation matrix W of the hidden state of the second model h , the transformation matrix W of the input feature x , bias term b, attention score vector V, input disk line feature data vector x at the current time step t t , the transpose of the attention score vector V T ;

[0176] By formula: α t =softmax(V T tanh(W h h t +W x x t +b)), calculate the attention weight α t ;

[0177] By formula: Calculate the context vector c;

[0178] Get the second model output layer weight W o and the second model output layer bias b o ;

[0179] By formula: z = W o c+b o , calculate the second model linear output z;

[0180] By formula: Calculate the exact risk probability value P for each disk in the first risk disk set Bi-LSTM ;

[0181] Determine whether the second risk probability value (accurate risk probability value) of any disk is less than a first preset value (30%);

[0182] If yes, an alarm is issued for the disk; if no, the disk is classified into the second risk disk set;

[0183] Among them, the second model is the bidirectional long short-term memory network model Bi-LSTM (Bidirectional LongShort-Term Memory).

[0184] Specifically, the focus loss function is improved: the modified FocalLoss is adopted:

[0185] FL(p t )=-α t (1-p t ) γ log(p t )

[0186] p t: model prediction probability;

[0187] γ: focusing parameter (usually 2);

[0188] α t : Category weight

[0189] Integrate the FocalLoss function into the model training process;

[0190] Adjust α and γ parameters based on SMART data;

[0191] Compare the effects of FocalLoss and ordinary cross entropy to achieve an end-to-end training + deployment solution;

[0192] Clipping the gradient: Prevent gradient explosion;

[0193]

[0194] θ: threshold (default is 5);

[0195] grad: the currently calculated gradient vector;

[0196] ||grad||2: L2 norm of the gradient vector;

[0197] max(1,||grad||2 / θ) : If the L2 norm of the gradient is greater than θ, scale the gradient using ||grad||2 / θ as the denominator; otherwise, keep it as it is.

[0198] By limiting the maximum norm of the gradient, the training process can be stabilized to avoid problems such as excessive parameter updates due to excessive gradients, which in turn cause model instability or divergence.

[0199] Step S05, as Figure 2 As shown, the risk level of the second risk disk set is divided according to the third model, system load and system disk redundancy, and the target control strategy corresponding to the disk is determined according to the disk risk level.

[0200] Step S051, obtaining the current state s of the function, the current state of the function including the second risk probability value of each disk in the second risk disk set, the system load, and the system disk redundancy;

[0201] Get the currently selected action a, the immediate reward t obtained after executing the action, the next state s' after executing the action, the next selected action a', the learning rate α, the discount factor γ, and the maximum Q value among all possible actions a' in the next state s'

[0202] Update the action-value function Q through the deep neural network of the third model:

[0203] Among them, the third model is the deep neural network model DQN (Deep Q-Network);

[0204] The second risk disk set is divided into risk levels according to the learning result of the action-value function;

[0205] When the precise risk probability value of the disk reaches the first threshold (risk probability 30-60%), the failure risk level of the disk is set to the warning level;

[0206] When the precise risk probability value of the disk is the second threshold (risk probability 60-85%), the failure risk level of the disk is set to a high-risk level;

[0207] When the precise risk probability value of the disk reaches the third threshold (risk probability>85%), the failure risk level of the disk is set to the emergency level;

[0208] Determine the target control strategy corresponding to the disk based on the disk risk level.

[0209] Specifically, the DQN layer uses a deep reinforcement learning algorithm to generate the optimal control isolation strategy; the core of DQN is the Q-Learning algorithm, which approximates the Q function (action value function) through a deep neural network.

[0210] (1) Q-Learning update rule:

[0211]

[0212] Function meaning: Update the action-value function Q(s,a) to approximate the immediate reward of the current action plus the maximum expected discounted return of the subsequent state; this update rule is used to train the parameters of the neural network (called the online network).

[0213] s: current status (including the precise risk probability output by the Bi-LSTM model, system load, RAID redundancy, etc.);

[0214] a: The currently selected action (such as speed reduction, hot migration, isolation, etc.);

[0215] t: the immediate reward obtained after performing the action (calculated by the reward function);

[0216] s': the next state entered after executing the action;

[0217] α: learning rate (controls update step size);

[0218] γ: discount factor (measures the importance of future rewards);

[0219] The maximum Q value among all possible actions a' in the next state s';

[0220] (2) Using dual network architecture:

[0221] θ - ←τθ+(1-τ)θ -

[0222] In order to stabilize the training of the DQN model, two neural networks are used:

[0223] Online network parameter θ: used to calculate the current Q value Q(s,a;θ)

[0224] Target network parameters θ - : Used to calculate the target Q value

[0225] (3) Experience replay:

[0226]

[0227] DQN uses an experience replay buffer to store transfer samples (s, a, r, s') and randomly trains from them to break the correlation of samples and improve stability;

[0228] p i =|δ i |+ε

[0229] in, δ i is the TD error; the importance of the sample is determined according to the absolute value of the TD error for priority playback;

[0230] p i =|δ i |+∈: TD error priority;

[0231] β: Importance sampling coefficient (increases linearly from 0.4 to 1.0)

[0232] Action selection strategy: Annealing formula for ∈-greedy strategy

[0233]

[0234] ∈ start =1.0,∈ end =0.01,∈ decay= 1000 (decay steps), the current time step or iteration number t; this exponential decay method can smoothly transition from the exploration phase to the utilization phase, which helps improve learning efficiency and final performance;

[0235] (4) Action selection strategy

[0236] During model training, an ∈-greedy strategy is used to balance exploration (trying new actions) and exploitation (selecting known optimal actions).

[0237] The DQN model uses technologies such as Experience Replay and Target Network to improve model training stability and efficiency.

[0238] It is understandable that if Figure 2 As shown in the figure, the intelligent perception layer obtains the SMART parameters, system performance indicators, and environmental data of each hard drive. The stream processing layer uses a distributed data processing pipeline built on Apache Flink to convert raw data into high-level features. The converted data is then imported into the hybrid model layer, where a three-level progressive model architecture accurately assesses the health status of each hard drive, enabling intelligent management of hard drive health.

[0239] L1: LightGBM layer: performs an initial risk assessment on the hard disk health status and obtains an initial risk probability value. Hard disks with an initial risk probability value less than 0.3 are considered normal and healthy; otherwise, they are considered abnormal disks with risks and enter the Bi-LSTM layer.

[0240] L2: Bi-LSTM layer: This layer performs in-depth and precise risk assessment on abnormal disks with risks, obtaining precise risk probability values. For abnormal disks with precise risk probability values ​​less than 0.3, a log is recorded and an alarm is issued. Otherwise, abnormal disks with higher risks are considered abnormal disks and enter the DQN layer.

[0241] L3: DQN layer: Actively handle abnormal disks with higher risks. Based on the conclusions of L2 and the judgment of system status (such as load and RAID redundancy), the optimal warning / high-risk / emergency three-level disk isolation strategy is generated.

[0242] Step S06: performing fault processing on the disk according to the target control policy corresponding to the disk.

[0243] Step S061: When the disk is at the warning level, the operation frequency of the disk is reduced or the workload of the disk is lowered, and important data is migrated from the warning level disk to a healthy storage device;

[0244] When a disk is classified as a high-risk disk, all data on the disk will be migrated to a backup storage device, and new data will be prevented from being written to the high-risk disk.

[0245] When a disk is classified as an emergency disk, all read and write operations on the disk are stopped, the disk is removed from the current storage pool, and the affected data in the storage system is restored based on the existing redundancy information.

[0246] Here, as Figure 3 As shown in this application, Step 1: Data Collection:

[0247] Intelligent sensing layer: collects SMART data and performance data from multiple hard drives (hard drives 1 to n), and also collects environmental data such as temperature, current, and vibration from sensors;

[0248] Kafka message queue: All collected data is sent to the Kafka message queue as the data source for subsequent processing;

[0249] Step 2: Streaming:

[0250] Flink real-time processing engine: reads data from the Kafka message queue and processes it in real time;

[0251] Feature calculation:

[0252] Calculate time domain features such as mean, kurtosis, and Hurst exponent;

[0253] Compute frequency domain features, such as wavelet energy spectrum;

[0254] Compute topological features such as rack dependencies;

[0255] Step 3: Mixing model layers:

[0256] L1: LightGBM quickly filters normal disks: Use the LightGBM model to preliminarily filter the data after feature calculation to filter out the data of normal hard disks;

[0257] Abnormal data processing:

[0258] For data marked as abnormal, the L2:Bi-LSTM risk probability prediction model is further used for risk assessment;

[0259] Based on the output of the L2 model, for high-risk data, the L3:DQN generation control strategy model is used to formulate the corresponding control strategy;

[0260] Step 4: Implementation and Feedback:

[0261] Instruction distribution: Based on the control strategy generated by the DQN model, the system will issue different warnings or control instructions:

[0262] General alarm: recorded in the log;

[0263] Important warning: It may be necessary to take measures such as speed reduction, migration, or isolation;

[0264] Emergency alarm: Operation and maintenance personnel are required to intervene immediately;

[0265] Model online update: Based on the execution results and feedback, the system will update the models in the hybrid model layer online to continuously optimize the accuracy of prediction and control strategies.

[0266] Through real-time data collection, streaming processing, hybrid model prediction, and dynamic execution and feedback mechanisms, early warning and effective management of hard disk failures are achieved.

[0267] In addition, the method further includes: performing layered filtering processing on the SMART raw data:

[0268] Perform threshold checks on the hard drive's SMART data through threshold checks (L1 level);

[0269] If the raw SMART value exceeds the manufacturer's specification range, the data point is considered abnormal. Once a data point exceeding the threshold is found, it is immediately marked for further analysis or appropriate measures.

[0270] Calculate the Z-Score of a data point relative to its moving average through the Moving Average Z-Score (L2 level);

[0271] According to the 3σ principle, if the absolute value of the Z-Score of a SMART data point is greater than 3, the SMART data point is considered a suspicious event;

[0272] SMART data points marked as suspicious events are reported to the system administrator or relevant person in charge for further investigation and confirmation;

[0273] SMART data were analyzed using the lightweight version of the Isolation Forest algorithm (L3 level).

[0274] If the anomaly probability of a SMART data point calculated by the isolation forest algorithm is greater than 0.7, the data point is considered to be highly likely an anomaly;

[0275] Trigger the deep analysis process to conduct deeper diagnosis and analysis on suspected abnormal SMART data points to determine whether there is indeed a problem and the specific nature and cause of the problem.

[0276] Through the above three steps, multi-level anomaly detection can be effectively performed on the hard disk SMART raw data, thereby improving the accuracy and reliability of disk failure anomaly detection.

[0277] The disk failure handling method provided in the embodiment of the present application can be further improved and optimized without departing from the technical solution of the present application, and these improvements and optimizations should also be considered as the scope of protection of the present application.

[0278] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0279] The technology of this application can effectively perform multi-level anomaly detection on disk failures, thereby improving the accuracy and reliability of disk failure processing.

[0280] The technical solution of this application 1. effectively improves the disk failure prediction performance: the fault detection rate reaches 92.4% (the related technology is 63.1%); the false alarm rate is reduced to 5.2% (the related technology is 28.7%); the average warning advance time reaches 37 hours; 2. optimizes the disk failure operation and maintenance efficiency: the fault handling time is shortened by 68%; the unexpected downtime is reduced by 83%; the hard disk replacement cost is reduced by 42%; 3. The system reliability is enhanced: the disk redundant array reconstruction failure rate is reduced from 15% to 2%; the data loss incidents are reduced by 91%; the storage system availability reaches 99.9%.

[0281] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0282] The embodiment of the present application also provides a disk fault processing device, such as Figure 4 As shown, the device includes: an acquisition module, an extraction module, a first filtering module, a second filtering module, a determination module, and a processing module.

[0283] In this embodiment, the acquisition module is used to obtain the operating status data of each disk; obtain the system load and the system disk redundancy;

[0284] The extraction module is used to extract line features from the operating status data of each disk. The line features include time domain features, frequency domain features, and topological features.

[0285] A first filtering module is configured to perform a first filtering on the disk operation status data after all line features are extracted using a first model to obtain a first risk disk set;

[0286] a second filtering module, configured to perform a second filtering on the first risk disk set using a second model to obtain a second risk disk set;

[0287] a determination module, configured to classify the risk level of the second risk disk set by using the third model, the system load, and the system disk redundancy, and determine the target control strategy corresponding to the disk according to the disk risk level;

[0288] The processing module is used to process disk failures according to a target control strategy corresponding to the disk.

[0289] In this embodiment, the acquisition module is used to collect the operating status data of each disk;

[0290] The disk operation status data includes disk health parameter data, disk performance data, and system environment data;

[0291] Disk performance data includes disk input / output latency, read / write throughput, and load rate;

[0292] System environment data includes ambient temperature and vibration frequency;

[0293] The operating status data of each disk is stored in the message queue.

[0294] In one embodiment, the extraction module is used to calculate the time domain features of the disk's operating status data by a sliding window method;

[0295] Use the Welford online algorithm to calculate the mean and variance of disk operation status data;

[0296] The kurtosis of disk operation status data is calculated by the fourth-order central moment;

[0297] The long-range correlation of disk operation status data was evaluated by rescaled range analysis;

[0298] The frequency domain characteristics of the disk's operating status data are extracted by wavelet transform, and orthogonal wavelet is set as the mother wavelet;

[0299] Calculating the energy spectrum of the disk operation status data according to the sum of squares of detail coefficients of the disk operation status data;

[0300] The topological characteristics of the disk's operating status data are calculated using the continuous coherence method, and the operating status correlation between different disks in the same rack is analyzed.

[0301] In one embodiment, the first filtering module is used to obtain the gradient boosting decision tree prediction function of the first model

[0302]

[0303] Where K is the total number of gradient boosting decision trees, f kis the prediction function of the k-th tree, For all possible decision tree spaces, x i is the input line feature vector data of the i-th disk; the first model is a gradient boosting decision tree model;

[0304] By formula: Calculate the first risk probability value P for each disk LightGBM , σ is the activation function;

[0305] Determining whether a first risk probability value of any disk is less than a first preset value;

[0306] If so, the disk is a normal disk and the process ends; if not, the disk is classified into the first risk disk set.

[0307] In one embodiment, the second filtering module is used to obtain the input line feature vector data x of the i-th disk in the first risk disk set. i , the forward weight matrix W → , the forward hidden state vector of the second model at the previous time step t

[0308] By formula: Calculate the second model forward hidden state vector

[0309] Get the inverse weight matrix W ← , the reverse hidden state vector of the second model at the next time step t

[0310] By formula: Calculate the second model reverse hidden state vector

[0311] By formula: Calculate the hidden state vector h of the second model t ;

[0312] Get the transformation matrix W of the hidden state of the second model h , the transformation matrix W of the input feature x , bias term b, attention score vector V, input disk line feature data vector x at the current time step t t , the transpose of the attention score vector V T ;

[0313] By formula: α t =softmax(V T tanh(W h h t +W x x t+b)), calculate the attention weight α t ;

[0314] By formula: Calculate the context vector c;

[0315] Get the second model output layer weight W o and the second model output layer bias b o ;The second model is a bidirectional long short-term memory network model;

[0316] By formula: z = W o c+b o , calculate the second model linear output z;

[0317] By formula: Calculate the second risk probability value P for each disk in the first risk disk set Bi-LSTM ;

[0318] Determining whether a second risk probability value of any disk is less than a first preset value;

[0319] If yes, an alarm is issued for the disk; if no, the disk is classified into the second risk disk set.

[0320] In one embodiment, the determination module is configured to obtain a current state s of the function, the current state of the function including a second risk probability value of each disk in the second risk disk set, a system load, and a system disk redundancy;

[0321] Get the currently selected action a, the immediate reward r after executing the action, the next state s' after executing the action, the next selected action a', the learning rate α, the discount factor γ, and all possible actions a in the next state s' , The maximum Q value

[0322] Update the action-value function Q through the deep neural network of the third model:

[0323] Among them, the third model is a deep neural network model;

[0324] The second risk disk set is divided into risk levels according to the learning result of the action-value function;

[0325] When the second risk probability value of the disk is the first threshold, the failure risk level of the disk is set to the warning level;

[0326] When the second risk probability value of the disk is the second threshold, the failure risk level of the disk is set to a high risk level;

[0327] When the second risk probability value of the disk is the third threshold, the failure risk level of the disk is set to the emergency level;

[0328] Determine the target control strategy corresponding to the disk based on the disk risk level.

[0329] In one embodiment, the processing module is configured to reduce the frequency of operations on the disk or reduce the workload of the disk when the disk is at the warning level, and migrate important data from the warning level disk to a healthy storage device;

[0330] When a disk is classified as a high-risk disk, all data on the disk will be migrated to a backup storage device, and new data will be prevented from being written to the high-risk disk.

[0331] When a disk is classified as an emergency disk, all read and write operations on the disk are stopped, the disk is removed from the current storage pool, and the affected data in the storage system is restored based on the existing redundancy information.

[0332] Specifically, this application collects multi-dimensional monitoring data (SMART, performance, environment) in real time;

[0333] Calculate the fault risk value through the LSTM-DQN hybrid model;

[0334] Implement hierarchical isolation strategies based on risk levels:

[0335] Warning level: Reduce I / O priority and start background scanning;

[0336] High-risk level: Limit write bandwidth and migrate hot data;

[0337] Emergency level: Forces the hard disk offline and triggers a rapid rebuild;

[0338] Among them, the distributed data acquisition agent is used to collect, preprocess and transmit data automatically, efficiently and scalably; it is usually composed of multiple agents distributed on different physical / virtual nodes, working together to complete large-scale data acquisition tasks.

[0339] Streaming feature processing pipeline: used to continuously receive, calculate, and output features to support machine learning models, real-time analysis, or decision-making systems; emphasizes low-latency, high-throughput processing capabilities and is suitable for dynamically changing data (such as user behavior, sensor data, transaction streams, etc.).

[0340] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0341] The technology of this application can effectively perform multi-level anomaly detection on disk failures, thereby improving the accuracy and reliability of disk failure processing.

[0342] The technical solution of this application 1. effectively improves the disk failure prediction performance: the fault detection rate reaches 92.4% (the related technology is 63.1%); the false alarm rate is reduced to 5.2% (the related technology is 28.7%); the average warning advance time reaches 37 hours; 2. optimizes the disk failure operation and maintenance efficiency: the fault handling time is shortened by 68%; the unexpected downtime is reduced by 83%; the hard disk replacement cost is reduced by 42%; 3. The system reliability is enhanced: the disk redundant array reconstruction failure rate is reduced from 15% to 2%; the data loss incidents are reduced by 91%; the storage system availability reaches 99.9%.

[0343] For the description of the features in the embodiment corresponding to the disk failure processing device, reference can be made to the relevant description of the embodiment corresponding to the disk failure processing method, which will not be repeated here.

[0344] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps of an embodiment of a disk failure processing method, the method comprising:

[0345] Obtain the operating status data of each disk; obtain the system load and system disk redundancy;

[0346] Extract line features from the operating status data of each disk. Line features include time domain features, frequency domain features, and topological features.

[0347] Performing a first filtering on the disk operation status data after all line features are extracted using the first model to obtain a first risk disk set;

[0348] Performing a second filtering on the first risk disk set using the second model to obtain a second risk disk set;

[0349] The risk level of the second risk disk set is divided according to the third model, the system load and the system disk redundancy, and the target control strategy corresponding to the disk is determined according to the disk risk level;

[0350] Handle disk failures according to the target control policy corresponding to the disk.

[0351] like Figure 5 As shown, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the steps of the disk failure processing method embodiment when running, the method comprising:

[0352] Obtain the operating status data of each disk; obtain the system load and system disk redundancy;

[0353] Extract line features from the operating status data of each disk. Line features include time domain features, frequency domain features, and topological features.

[0354] Performing a first filtering on the disk operation status data after all line features are extracted using the first model to obtain a first risk disk set;

[0355] Performing a second filtering on the first risk disk set using the second model to obtain a second risk disk set;

[0356] The risk level of the second risk disk set is divided according to the third model, the system load and the system disk redundancy, and the target control strategy corresponding to the disk is determined according to the disk risk level;

[0357] Handle disk failures according to the target control policy corresponding to the disk.

[0358] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0359] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the disk failure processing method embodiment are implemented. The method includes:

[0360] Obtain the operating status data of each disk; obtain the system load and system disk redundancy;

[0361] Extract line features from the operating status data of each disk. Line features include time domain features, frequency domain features, and topological features.

[0362] Performing a first filtering on the disk operation status data after all line features are extracted using the first model to obtain a first risk disk set;

[0363] Performing a second filtering on the first risk disk set using the second model to obtain a second risk disk set;

[0364] The risk level of the second risk disk set is divided according to the third model, the system load and the system disk redundancy, and the target control strategy corresponding to the disk is determined according to the disk risk level;

[0365] Handle disk failures according to the target control policy corresponding to the disk.

[0366] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing the steps in the disk failure processing method embodiment, the method including:

[0367] Obtain the operating status data of each disk; obtain the system load and system disk redundancy;

[0368] Extract line features from the operating status data of each disk. Line features include time domain features, frequency domain features, and topological features.

[0369] Performing a first filtering on the disk operation status data after all line features are extracted using the first model to obtain a first risk disk set;

[0370] Performing a second filtering on the first risk disk set using the second model to obtain a second risk disk set;

[0371] The risk level of the second risk disk set is divided according to the third model, the system load and the system disk redundancy, and the target control strategy corresponding to the disk is determined according to the disk risk level;

[0372] Handle disk failures according to the target control policy corresponding to the disk.

[0373] The technology of this application can effectively perform multi-level anomaly detection on disk failures, thereby improving the accuracy and reliability of disk failure processing.

[0374] The technical solution of this application 1. effectively improves the disk failure prediction performance: the fault detection rate reaches 92.4% (the related technology is 63.1%); the false alarm rate is reduced to 5.2% (the related technology is 28.7%); the average warning advance time reaches 37 hours; 2. optimizes the disk failure operation and maintenance efficiency: the fault handling time is shortened by 68%; the unexpected downtime is reduced by 83%; the hard disk replacement cost is reduced by 42%; 3. The system reliability is enhanced: the disk redundant array reconstruction failure rate is reduced from 15% to 2%; the data loss incidents are reduced by 91%; the storage system availability reaches 99.9%.

[0375] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0376] The above describes in detail the disk failure handling method, apparatus, device, and medium provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is intended only to help understand the method and core concept of the present application. It should be noted that, for those skilled in the art, various improvements and modifications may be made to the present application without departing from the principles of the present application, and such improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A disk failure processing method, characterized in that: The method is applied to a server storage system, the system including several disks, and the method includes: Obtain the operating status data of each disk, the system load, and the system disk redundancy; Extracting line features from the operating status data of each disk, wherein the line features include time domain features, frequency domain features, and topological features; Performing a first filtering on the disk operation status data after all line features are extracted using the first model to obtain a first risk disk set; Performing a second filtering on the first risk disk set using the second model to obtain a second risk disk set; The risk level of the second risk disk set is divided according to the third model, the system load and the system disk redundancy, and the target control strategy corresponding to the disk is determined according to the disk risk level; Perform disk fault processing according to the target control policy corresponding to the disk.

2. The disk failure processing method according to claim 1, characterized in that: The step of obtaining the operating status data of each disk includes: Collect the operating status data of each disk; The disk operation status data includes disk health parameter data, disk performance data and system environment data; The disk performance data includes disk input / output latency, read / write throughput, and load rate; The system environment data includes ambient temperature and vibration frequency; The operating status data of each disk is stored in a message queue.

3. The disk failure processing method according to claim 1, characterized in that: The extracting of line features from the operating status data of each disk includes: Calculating the time domain characteristics of the disk's operating status data by a sliding window method; Use the Welford online algorithm to calculate the mean and variance of disk operation status data; The kurtosis of disk operation status data is calculated by the fourth-order central moment; The long-range correlation of disk operation status data was evaluated by rescaled range analysis; Extracting frequency domain features of the disk's operating status data by wavelet transform, and setting an orthogonal wavelet as a mother wavelet; Calculating an energy spectrum of the disk operation status data according to the sum of squares of detail coefficients of the disk operation status data; The topological characteristics of the operating status data of the disk are calculated by a continuous coherence method, and the operating status correlation between different disks in the same rack is analyzed.

4. The disk failure processing method according to claim 1, wherein: The first model is used to perform a first filtering on the disk operation status data after all line features are extracted to obtain a first risk disk set, including: Get the gradient boosting decision tree prediction function of the first model Where K is the total number of gradient boosting decision trees, f k is the prediction function of the k-th tree, For all possible decision tree spaces, x i is the input line feature vector data of the i-th disk; the first model is a gradient boosting decision tree model; By formula: Calculate the first risk probability value P for each disk LightGBM , σ is the activation function; Determining whether a first risk probability value of any disk is less than a first preset value; If so, the disk is a normal disk and the process ends; if not, the disk is classified into the first risk disk set.

5. The disk failure processing method according to claim 4, characterized in that: The second filtering of the first risk disk set by the second model to obtain a second risk disk set includes: Get the input line feature vector data x of the i-th disk in the first risk disk set i , the forward weight matrix W → , the forward hidden state vector of the second model at the previous time step t By formula: Calculate the second model forward hidden state vector Get the inverse weight matrix W ← , the reverse hidden state vector of the second model at the next time step t By formula: Calculate the second model reverse hidden state vector By formula: Calculate the hidden state vector h of the second model t ; Get the transformation matrix W of the hidden state of the second model h , the transformation matrix W of the input feature x , bias term b, attention score vector V, input disk line feature data vector x at the current time step t t , the transpose of the attention score vector V T ; By formula: α t =softmax(V T tanh(W h h t +W x x t +b)), calculate the attention weight α t ; By formula: Calculate the context vector c; Get the second model output layer weight W o and the second model output layer bias b o ; The second model is a bidirectional long short-term memory network model; By formula: z = W o c+b o , calculate the second model linear output z; By formula: Calculate the second risk probability value P of each disk in the first risk disk set Bi-LSTM ; Determining whether a second risk probability value of any disk is less than a first preset value; If yes, an alarm is issued for the disk; if no, the disk is classified into the second risk disk set.

6. The disk failure processing method according to claim 5, characterized in that: The risk level classification of the second risk disk set is performed based on the third model, the system load, and the system disk redundancy, and the target control strategy corresponding to the disk is determined according to the disk risk level, including: Obtaining a current state s of the function, wherein the current state of the function includes a second risk probability value of each disk in the second risk disk set, a system load, and a system disk redundancy; Get the currently selected action a, the immediate reward r after executing the action, the next state s' after executing the action, the next selected action a', the learning rate α, the discount factor γ, and the maximum Q value among all possible actions a' in the next state s' Update the action-value function Q through the deep neural network of the third model: Wherein, the third model is a deep neural network model; Classifying the second risk disk set into risk levels according to the learning result of the action-value function; When the second risk probability value of the disk is the first threshold, setting the failure risk level of the disk to a warning level; When the second risk probability value of the disk is a second threshold, the failure risk level of the disk is set to a high risk level; When the second risk probability value of the disk is the third threshold, the failure risk level of the disk is set to an emergency level; Determine the target control strategy corresponding to the disk based on the disk risk level.

7. The disk failure processing method according to claim 6, characterized in that: The performing disk fault processing according to the target control strategy corresponding to the disk includes: When the disk is at the warning level, the operation frequency of the disk is reduced or the workload of the disk is lowered, and important data is migrated from the warning level disk to a healthy storage device; When the disk is a high-risk disk, all data on the disk is migrated to a backup storage device, and new data is prevented from being written to the high-risk disk; When the disk is an emergency-level disk, all read and write operations on the disk are stopped, the disk is removed from the current storage pool, and the affected data of the storage system is restored based on the existing redundancy information.

8. A disk failure processing device, characterized in that: The device comprises: The acquisition module is used to obtain the operating status data of each disk; obtain the system load and system disk redundancy; An extraction module is used to extract line features from the operating status data of each disk, wherein the line features include time domain features, frequency domain features, and topological features; A first filtering module is configured to perform a first filtering on the disk operation status data after all line features are extracted using a first model to obtain a first risk disk set; a second filtering module, configured to perform a second filtering on the first risk disk set using a second model to obtain a second risk disk set; a determination module, configured to classify the risk level of the second risk disk set by using the third model, the system load, and the system disk redundancy, and determine the target control strategy corresponding to the disk according to the disk risk level; The processing module is used to perform fault processing on the disk according to the target control strategy corresponding to the disk.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the disk failure handling method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the disk failure processing method according to any one of claims 1 to 7 are implemented.

Images