Method for establishing solid state disk failure prediction model based on abnormal perception hierarchical sampling and application thereof

Abstract

The application discloses a solid state disk fault prediction model establishment method based on abnormal perception hierarchical sampling and application thereof, and belongs to the technical field of solid state disk fault prediction, and comprises the following steps: constructing a fault prediction sample based on a solid state disk fault time and a prediction lead time, implementing hierarchical sampling on a training sample according to the number of SMART attributes that simultaneously appear abnormal changes in a single sampling period, retaining difficult-to-classify samples close to a decision boundary while compressing redundant health samples; in view of a fault feature dilution problem, constructing multi-scale time sequence features including a difference, a degradation slope and a trend acceleration, and enhancing feature separability of a fault disk and a health disk; and in a deployment stage, determining an alarm threshold and outputting a fault risk result under the condition of a device-level false alarm rate constraint. The application improves recall capability and deployability of solid state disk fault prediction under a low false alarm constraint on the premise of only relying on general SMART attributes, and can provide early warning basis for data migration, device replacement and risk disposal.

Description

Technical Field

[0001] This invention belongs to the field of solid-state drive (SSD) fault prediction technology, and more specifically, relates to a method for establishing an SSD fault prediction model based on anomaly perception and hierarchical sampling, and its application. Background Technology

[0002] Solid-state drives (SSDs), with their high random read / write performance, low access latency, and low power consumption, have become the primary storage medium in data centers, cloud computing platforms, enterprise-level storage systems, and server clusters. To ensure data reliability, existing data centers typically employ passive redundancy fault-tolerance mechanisms such as independent disk redundant arrays and erasure coding, restoring services through data reconstruction after a single disk failure. However, as the capacity of individual disks continues to increase, data reconstruction after a failure can trigger sustained high-bandwidth read / write pressure, consuming network and computing resources, and even inducing concurrent failures of other devices within the same array or rack. Therefore, relying solely on passive fault-tolerance mechanisms that rely on post-failure reconstruction is no longer sufficient to meet the stability, operational cost, and service continuity requirements of large-scale storage systems.

[0003] SMART (Self-Monitoring, Analysis and Reporting Technology) is a built-in health monitoring system for hard drives. It predicts faults and protects data security by monitoring parameters such as temperature, circuitry, and media. Proactive fault prediction based on SMART attributes can utilize health monitoring data continuously generated by the SSD during operation, such as power-on time, write cycles, error counts, temperature, uncorrectable errors, and read retry counts, to identify high-risk SSDs before irreversible failures occur. This assists the maintenance system in proactively migrating data, replacing equipment, or handling risks. Compared to solutions relying on vendor-proprietary hardware logs, SMART attributes offer advantages such as low data collection costs, universal interfaces, and low deployment barriers, making it the most engineering-usable input data source for SSD health monitoring and fault prediction systems.

[0004] However, existing fault prediction methods based on SMART attributes face significant shortcomings in industrial deployment conditions. First, in real-world production environments, healthy samples far outnumber faulty samples, with the ratio of positive to negative samples often reaching thousands of times. The classifier training process is easily dominated by the massive number of healthy samples, causing the decision boundary to shift towards faulty samples, making it difficult to effectively identify a few faulty samples. Second, key SMART attributes preceding SSD failures often exhibit short-term mutations or accelerated degradation within a few days. Coarse-grained statistical features such as long-window moving averages, maximum values, and minimum values ​​smooth out these short-term mutations, diluting key fault signals before they enter the model. Third, large-scale storage systems have strict constraints on false alarm rates. Excessively high device-level false alarm rates generate a large number of invalid alarms, causing unnecessary data migration, equipment replacement, and alarm fatigue. Under low false alarm rate constraints, many models, even those performing well on conventional classification metrics, are difficult to translate into deployable high fault recall rates.

[0005] Existing technologies for SSD fault prediction can be broadly categorized into three approaches: One approach is based on general SMART attributes and traditional machine learning models. These methods are easy to deploy and highly interpretable, but often rely on single-point snapshots or coarse-grained statistical features, making it difficult to capture short-term dynamic changes before a fault. Another approach is based on deep time-series models such as Long Short-Term Memory networks and Transformers. These methods have strong time-series fitting capabilities, but suffer from high computational overhead, weak interpretability, and may not be suitable for lightweight deployments in industrial settings. A third approach integrates firmware internal logs, underlying error channels, or vendor-specific hardware features. While this may improve prediction performance, it has high data access barriers, limiting its practical deployment scope.

[0006] Therefore, a new method for predicting solid-state drive (SSD) failures is needed to effectively mitigate the decision boundary shift caused by extreme class imbalances and improve the reliability and accuracy of SSD failure prediction, while relying solely on the general SMART attribute. Summary of the Invention

[0007] To address the shortcomings and improvement needs of existing technologies, this invention provides a method for establishing a solid-state drive (SSD) fault prediction model based on anomaly perception hierarchical sampling and its application. The aim is to effectively alleviate the decision boundary offset caused by extreme class imbalance and improve the reliability and accuracy of SSD fault prediction by relying solely on the general SMART attribute.

[0008] To achieve the above objectives, according to one aspect of the present invention, a method for establishing a solid-state drive fault prediction model based on anomaly perception hierarchical sampling is provided, comprising: S1: Collect SMART attribute timing data of each solid-state drive according to the preset sampling period, and record the status of the solid-state drive at each sampling time. Mark the SMART attribute record within the preset fault warning window before the fault time as a fault sample, and mark the SMART attribute record before the fault warning window as a healthy sample. S2: Count the number of SMART attributes that show abnormal changes in each healthy sample relative to the previous sampling time. This number is used as the short-term abnormal concurrency of the healthy samples. The healthy sample set is then divided into groups with a short-term abnormal concurrency greater than [a certain value]. The adversarial hard negative sample set and short-term abnormal concurrency belong to A high-risk healthy sample set and a short-term abnormal concurrency level of less than A low-risk healthy sample set; ; S3: According to the first retention ratio respectively Second retention ratio The high-risk health sample set and the low-risk health sample set are sampled, and the sampled health samples, adversarial hard negative sample set and fault samples, together with the corresponding state labels, constitute the prediction sample set. ; S4: Divide the prediction sample set into training set, validation set and test set according to the preset ratio. Use the training set to train the machine learning model to obtain the solid-state drive failure prediction model for predicting the probability of solid-state drive failure based on SMART attributes. Use the validation set and test set to verify and test the model respectively to complete the establishment of the solid-state drive failure prediction model.

[0009] Furthermore, between S3 and S4, the following is also included: Remove the SMART attributes corresponding to the manufacturer's normalized score from each sample, and select the SMART attributes that are most correlated with the health status of the solid-state drive from the remaining SMART attributes to obtain the basic attribute set; perform basic feature processing on the SMART attributes in the basic attribute set to obtain the sample feature set corresponding to each sample. Furthermore, in S4, when training, validating, and testing the solid-state drive failure prediction model using the training set, validation set, and test set, the sample feature set corresponding to each sample is used as the model input. Basic feature processing includes: Each SMART attribute is treated as a feature and added to the sample feature set; According to semantics, the SMART attribute, which uses composite coding logic, is split into multiple sub-features, and the resulting sub-features are also added to the sample feature set. Apply a logarithmic translation transformation to each feature in the sample feature set.

[0010] Furthermore, the basic attribute set includes abnormal candidate attributes that will undergo abrupt changes within a preset fault warning window before the fault occurs; Furthermore, between S3 and S4, the following is also included: Multi-scale temporal features are constructed for each anomaly candidate attribute in the basic attribute set and added to the sample feature set; Multi-scale temporal features include: difference features; The methods for constructing differential features include: Calculate the target sampling time attribute values Compared with its predecessor Each sampling time attribute values The result of the difference operation ; For the result of the difference operation By applying a quantile transformation based on the empirical cumulative distribution, the difference features are obtained; in, .

[0011] Furthermore, multi-scale temporal features also include: slope features; The construction methods for slope features include: At the target sampling time The attribute values ​​within the previously preset historical observation window were fitted with a linear degradation slope using the least squares method. With linear degradation slope As a slope feature.

[0012] Furthermore, multi-scale temporal features also include: speedup features; Methods for constructing speedup ratio features include: Calculate short-term rate and long-term rate and in accordance with Calculation speedup ; For acceleration ratio Applying a quantile transformation based on the empirical cumulative distribution yields the speedup characteristics; in, The length of the historical observation window, For the target sampling time attribute values Compared with its predecessor Each sampling time attribute values The result of the difference operation; This is a preset constant.

[0013] Furthermore, S4 also includes: during model training and validation, enabling the model to output the contribution of each feature to the prediction result, and updating the sample feature set through sample filtering; sample filtering includes: Features with a missing proportion higher than the preset removal threshold, or features whose values ​​remain unchanged for a duration exceeding the preset duration, are removed. For multiple features with a similarity higher than a preset similarity threshold, only the feature with the highest contribution is retained; Features whose contribution to the training set is higher than the preset contribution threshold, but whose contribution to the validation set is unstable, are removed.

[0014] Furthermore, S4 also includes: setting a candidate fault alarm threshold set, wherein the fault alarm threshold is used to determine whether the solid-state drive is about to fail; when the probability output by the solid-state drive prediction model is greater than or equal to the fault alarm threshold, it is determined that the solid-state drive is about to fail. For each fault alarm threshold in the candidate fault alarm threshold set, calculate the device-level false alarm rate and fault recall rate on the verification set, and add the fault alarm threshold with the corresponding device-level false alarm rate not exceeding the preset upper limit to the candidate deployment threshold set. The fault alarm threshold with the best fault recall rate is selected from the candidate deployment threshold set as the final fault alarm threshold.

[0015] According to another aspect of the present invention, a solid-state drive (SSD) fault prediction method based on anomaly perception hierarchical sampling is provided, comprising: inputting the SMART attribute of the SSD to be predicted into the SSD fault prediction model, obtaining the probability that the SSD will fail within a preset fault warning window before the prediction time, comparing it with a fault alarm threshold, and if it is greater than or equal to the fault alarm threshold, then predicting that the SSD to be predicted will fail. The solid-state drive (SSD) fault prediction model is established by the SSD fault prediction model establishment method based on anomaly perception hierarchical sampling provided by this invention.

[0016] Furthermore, the solid-state drive failure prediction method based on anomaly perception hierarchical sampling provided by the present invention also includes: generating feature attribution results when predicting that the solid-state drive to be predicted is about to fail; Feature attribution results include: the positive or negative contribution of each feature to the current prediction result.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, including a stored computer program; when the computer program is executed by a processor, it implements the solid-state disk fault prediction model establishment method based on anomaly perception hierarchical sampling provided by the present invention, and / or the solid-state disk fault prediction method based on anomaly perception hierarchical sampling provided by the present invention.

[0018] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) After constructing fault samples and healthy samples based on SMART attributes, this invention performs hierarchical sampling for anomaly perception. Specifically, based on the SMART attributes that show abnormal increments at the same sampling time, the abnormal concurrency of the samples is characterized. The healthy samples are further divided into adversarial hard negative samples, high-risk healthy samples and low-risk healthy samples. All fault samples and adversarial hard negative samples are retained. For high-risk healthy samples and low-risk samples, further sampling is performed. The retention ratio of high-risk healthy samples is even lower. Thus, while compressing a large number of redundant steady-state healthy samples, difficult-to-classify healthy samples that are valuable for learning the classification boundary are retained. This effectively alleviates the decision boundary shift caused by extreme class imbalance and ensures the prediction performance of the established model. Under the premise of relying only on general SMART attributes, the reliability and accuracy of solid-state disk fault prediction are improved.

[0019] (2) This invention further performs basic feature processing on the collected SMART attributes, including splitting the SMART attributes using composite coding logic into multiple sub-features according to semantics. This avoids the large cardinality of high-order bits masking the numerical fluctuations of small error counts in low-order bits when the SMART attributes using composite coding logic are input into the model, ensuring that the model can effectively distinguish the changes in various error techniques at the underlying level, further improving the performance of the model, as well as the reliability and accuracy of subsequent predictions. Applying a logarithmic translation transformation to each feature avoids extreme heavy-tailed distributions of numerical values ​​while strictly preserving the temporal monotonicity of the features and the relative ordering between samples.

[0020] (3) The present invention further constructs multi-scale temporal features for each anomaly candidate attribute in the SMART attribute, the multi-scale temporal features including the target sampling time. attribute values The difference calculation results of the attribute values ​​at the 1st, 3rd, and 7th sampling times. , and ,in, As a discrete approximation of the first derivative, it directly reflects the magnitude of newly added abnormal events in a single sampling period; and By preserving short-term trend information while smoothing out sampling jitter introduced by job scheduling during the sampling period, the model can more accurately capture the intrinsic relationship between SMART attributes and SSD health status.

[0021] (4) The multi-scale temporal features constructed by the present invention for abnormal candidate attributes also include slope features, that is, the target sampling time The linear degradation slope corresponding to the attribute values ​​within the previously preset historical observation window quantifies the average degradation rate of the device on a macro time scale, reflecting the long-term trend of attributes continuously deteriorating from normal levels to failure thresholds. Based on these features, the model can more accurately capture the intrinsic relationship between SMART attributes and the health status of the solid-state drive.

[0022] (5) The multi-scale time-series features constructed by the present invention for abnormal candidate attributes also include speedup features, which are calculated from short-term rate and long-term rate. Speedup features reflect the deterioration rate of recent attribute values ​​and provide additional indication information for the model to determine whether the device is in the accelerated degradation stage.

[0023] (6) The present invention further applies a quantile transformation based on empirical cumulative distribution to the difference operation results and speedup results, which can force the long-tailed distribution to be mapped to a uniform distribution of [0,1], thus suppressing the absolute magnitude while preserving the relative magnitude relationship of the time series features.

[0024] (7) The present invention performs sample filtering on the sample feature set composed of original attribute features, basic feature processing features and multi-scale time series features, and removes redundant features, low stability features and random features. Thus, while retaining multi-scale time series change information, it can reduce the interference of redundant and noisy features on model training.

[0025] (8) Instead of using a fixed default threshold, this invention sets a set of candidate fault alarm thresholds, calculates the device-level false alarm rate and fault recall rate one by one in the set of candidate fault alarm thresholds, and selects the threshold with the best fault recall rate among the thresholds where the device-level false alarm rate does not exceed the preset upper limit as the final threshold. Through this threshold determination method, the risk probability output by the model is transformed into a fault prediction result that meets the low false alarm deployment constraint.

[0026] (9) When predicting that a solid disk to be predicted is about to fail, the present invention further generates feature attribution results, including the positive or negative contribution of each feature to the current prediction result. These feature attribution results can be used as auxiliary information to help maintenance personnel identify the main abnormal attributes. Attached Figure Description

[0027] Figure 1 The flowchart illustrates the method for establishing a solid-state drive fault prediction model based on anomaly perception hierarchical sampling, as provided in this embodiment of the invention.

[0028] Figure 2 This is a schematic diagram of a method for establishing a solid-state drive fault prediction model based on anomaly perception hierarchical sampling, as provided in an embodiment of the present invention.

[0029] Figure 3This is a mapping diagram of SSD data stream processing path and fault prediction data source provided in an embodiment of the present invention.

[0030] Figure 4 This is a flowchart illustrating the sample construction and label generation process provided in an embodiment of the present invention.

[0031] Figure 5 This is a schematic diagram of anomaly perception hierarchical sampling provided in an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0033] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0034] To effectively address the technical problem that extreme class imbalance affects the reliability and accuracy of solid-state drive (SSD) failure prediction, this invention provides a method for establishing an SSD failure prediction model based on anomaly perception hierarchical sampling and its application.

[0035] This invention first constructs SMART time-series samples for early fault prediction. SMART attribute time-series data is collected during the operation of solid-state drives (SSDs). Historical operation logs of the same SSD are aggregated and sorted according to the device's unique identifier. Binary labels are generated based on the actual fault time and the predicted lead time, so that samples within a preset time window before the fault correspond to the fault warning state, while other valid operating samples correspond to the healthy state. This step provides a unified data foundation for subsequent sampling, feature construction, and model training.

[0036] In practical applications, SMART attributes are collected according to the approved sampling period to form SMART attribute time-series data. Changes in the SMART attributes of solid-state drives typically exhibit a daily periodicity; without loss of generality, the following embodiments use the day as the sampling period. Each sampling period can also be referred to as a sampling day.

[0037] To address the problem of extreme class imbalance, this invention designs an anomaly-aware hierarchical sampling mechanism based on the number of abnormal changes in multiple attributes within a single day, defining this number as the anomaly concurrency degree. This mechanism does not use simple randomness as the basis for selecting healthy samples in the first stage, but rather characterizes the anomaly concurrency degree of the samples based on the number of SMART attributes showing simultaneous abnormal increases within a single day. A higher anomaly concurrency degree indicates that the healthy sample is more likely to be close to the failure decision boundary. Therefore, the training samples are divided into subdomains such as failure samples, adversarial hard negative samples, high-risk healthy samples, and low-risk healthy samples, and differentiated retention strategies are set for different subdomains. This process compresses a large number of redundant steady-state healthy samples while retaining difficult-to-classify healthy samples that are valuable for learning the classification boundary.

[0038] To address the issue of short-term fault feature dilution, this invention establishes a mechanism for decoupling composite attributes and enhancing multi-scale temporal features. For composite SMART attributes containing multiple alarm bits, status bits, or subfields, bit-level decoupling is performed, and numerical transformations are applied to the original count values ​​and the decoupled attributes to form a more stable set of basic features. Based on this, multi-scale temporal features are constructed from three perspectives: short-term abrupt changes, medium-term degradation, and trend acceleration. These features include differential features, degradation slope features, and trend acceleration features under different time windows, replacing the compressed representation of fault dynamic signals by a single long-window statistical approach.

[0039] To address the model training and deployment decision-making issues, this invention employs robust feature selection based on a multi-scale feature set to reduce the impact of noisy, redundant, and locally spurious features on model training. Subsequently, a fault prediction model is trained to output the probability of a solid-state drive's failure risk at a given sampling time. During the deployment phase, this invention uses a preset threshold as a hard constraint to ensure that the device-level false alarm rate does not exceed a certain threshold. It searches for fault alarm thresholds on the validation set and selects the threshold with the highest recall rate from among the candidate thresholds that satisfy the false alarm constraint for deployment.

[0040] Through the above methods, this invention preserves difficult-to-distinguish samples near the decision boundary at the data distribution level, explicitly enhances signals of short-term abrupt changes, continuous degradation, and accelerated trends before failure at the feature representation level, and directly correlates model output with low false alarm rate constraints at the deployment decision level. Compared to fault prediction methods that rely solely on random sampling, single-point SMART snapshots, or long-window statistics, this invention can improve fault recall capabilities in low false alarm scenarios under general SMART attribute conditions, and provide more reliable early warning basis for data migration, equipment replacement, risk isolation, and preventative maintenance.

[0041] The following is an example.

[0042] Example 1: A method for establishing a solid-state drive fault prediction model based on anomaly perception hierarchical sampling, such as... Figure 1 and Figure 2 As shown, it includes: S1: Collect SMART attribute timing data of each solid-state drive according to the preset sampling period, and record the status of the solid-state drive at each sampling time. Mark the SMART attribute record within the preset fault warning window before the fault time as a fault sample, and mark the SMART attribute record before the fault warning window as a healthy sample. S2: Count the number of SMART attributes that show abnormal changes in each healthy sample relative to the previous sampling time. This number is used as the short-term abnormal concurrency of the healthy samples. The healthy sample set is then divided into groups with a short-term abnormal concurrency greater than [a certain value]. The adversarial hard negative sample set and short-term abnormal concurrency belong to A high-risk healthy sample set and a short-term abnormal concurrency level of less than A low-risk healthy sample set; ; S3: According to the first retention ratio respectively Second retention ratio The high-risk health sample set and the low-risk health sample set are sampled, and the sampled health samples, adversarial hard negative sample set and fault samples, together with the corresponding state labels, constitute the prediction sample set. ; S4: Divide the prediction sample set into training set, validation set and test set according to the preset ratio. Use the training set to train the machine learning model to obtain the solid-state drive failure prediction model for predicting the probability of solid-state drive failure based on SMART attributes. Use the validation set and test set to verify and test the model respectively to complete the establishment of the solid-state drive failure prediction model.

[0043] The following provides a further explanation of the specific implementation of each step.

[0044] In industrial deployments such as data centers or enterprise-level storage systems, a large number of solid-state drives (SSDs) are typically deployed, such as... Figure 3 As shown, the SMART attributes collected in this embodiment can originate from the operating stages such as the solid-state drive front-end interface, flash conversion layer, error correction and retry engine, and underlying flash media, and can reflect status information such as I / O latency, controller busy time, write amplification, read retry, error count, and media wear.

[0045] In step S1 of this embodiment, the specific construction of the fault prediction sample is as follows: Figure 4As shown, the system reads the SMART attribute timing data during the operation of the solid-state drive (SSD) and sorts the original records according to the SSD identifier and sampling time to form an operational status sequence for each SSD. Each operational record includes the SSD identifier, sampling time, several SMART attribute values, and corresponding device fault status information.

[0046] The system reads the SMART attribute timing data during the operation of the solid-state drive (SSD) and sorts the raw records according to the SSD identifier and sampling time to form an operational status sequence for each SSD. Each operational record includes the SSD identifier, sampling time, several SMART attribute values, and corresponding device fault status information.

[0047] like Figure 4 As shown, for a solid-state drive (SSD) that has already failed, a fault warning window is determined based on its actual failure time and a preset prediction lead time. If a sampling time falls before the actual failure time and within the fault warning window, the sample corresponding to that sampling time is marked as a faulty sample; if a sampling time falls outside the fault warning window, or if the SSD does not fail within the observation period, the corresponding sample is marked as a healthy sample. Operational records after a failure are not used as prediction samples in training to avoid interference from the post-failure state with the pre-failure prediction task.

[0048] After label construction is complete, the training, validation, and test sets are divided according to time sequence or device dimension, preventing the training process from utilizing future information from the testing phase. Through this step, the original SMART runtime logs are transformed into an initial set of prediction samples that can be used for supervised learning.

[0049] In solid-state drive (SSD) failure prediction tasks, the number of healthy samples is usually much higher than the number of failed samples. If the original training set is used directly to train the model, the classifier is easily dominated by the large number of healthy samples, thereby weakening its ability to identify the few failed samples. To alleviate the above problem, this embodiment does not use simple random undersampling as the basis for selecting healthy samples, but instead implements differential retention based on the density of abnormal information contained in the healthy samples.

[0050] First, through step S2, the abnormal concurrency of samples is characterized by the number of SMART attributes that show abnormal increases simultaneously within a single day. The healthy sample set is then divided into a high-risk healthy sample set and a low-risk healthy sample set, laying the foundation for hierarchical sampling for anomaly perception.

[0051] Specifically, the change of each SMART attribute between adjacent sampling days is calculated, and the number of SMART attributes that show abnormal changes within the same sampling day for the same solid-state drive is counted as the abnormal concurrency level. This abnormal concurrency level is then used as the basis for measuring the short-term abnormality of the sample.

[0052] The SMART attribute is defined as a healthy sample that shows an abnormal change (i.e., the change exceeds a corresponding threshold) relative to the previous sampling time. Specifically, it refers to the SMART attribute whose value changes more than a preset threshold between two adjacent sampling times. In this embodiment, when calculating the change of the SMART attribute between adjacent days, for cumulative attributes, the system focuses on its daily increment; for status or alarm attributes, the system focuses on its status change; and for attributes whose missing patterns are significant, the system retains the corresponding missing marker.

[0053] In step S2, this embodiment divides the samples into multiple levels. Based on this, such as Figure 5 As shown, S3 in this embodiment performs differentiated retention for different levels: the system retains fault samples within the fault warning window; for high-risk healthy samples with high abnormal concurrency, the retention ratio is increased so that difficult-to-distinguish healthy samples close to the feature space of fault samples can enter the training set; for low-risk healthy samples with low abnormal concurrency, which are usually ordinary healthy samples that are stable for a long time and have no obvious abnormal changes, the retention ratio is reduced in this embodiment to compress the scale of redundant majority class samples.

[0054] Through steps S2 and S3, this embodiment achieves hierarchical sampling for anomaly detection, and the resulting predicted sample set is used to compress the size of redundant majority class samples.

[0055] As a preferred implementation, this embodiment performs basic feature processing and constructs multi-scale temporal features before training the model using the prediction sample set.

[0056] In the raw dataset collected in this embodiment, each log entry has 105 columns, of which 51 columns are the original values ​​of the physical counters (in r_). (with n as a prefix), 51 columns are the manufacturer's normalized score (in n_ (with "as prefix"), the remaining 3 columns are metadata fields such as device identifier and timestamp.

[0057] Considering the opaque mapping logic and lack of cross-model consistency in the manufacturer's normalized score mapping across different models, this embodiment removes such SMART attributes and selects the SMART attributes most relevant to the health status of the SSD from the remaining SMART attributes to obtain a basic attribute set. The basic attribute set includes abnormal candidate attributes that will undergo abrupt changes (the change in a single sampling period exceeds a preset threshold) within a preset fault warning window before the failure time. In this embodiment, the basic attribute set specifically includes 12 abnormal candidate attributes and 9 auxiliary attributes used to describe wear background, workload, and environmental conditions: cumulative power-on time r_9, number of start-stop cycles r_12, wear leveling count r_173, number of abnormal power outages r_174, remaining spare blocks r_180, drive temperature r_194, number of hardware error corrections r_195, total host write volume r_241, and total host read volume r_242.

[0058] In the set of basic attributes, the anomaly candidate attributes are mainly used for short-term anomaly capture and serve as the core input of the multi-scale operator, which provides the model with auxiliary criteria for the device lifecycle stage and operating environment.

[0059] Of the 21 attributes mentioned above, five attributes—r_1, r_180, r_195, r_241, and r_242—possess vendor-specific composite encoding logic. Taking r_1 as an example, the high 16 bits of its original value record the total number of operations, and the low 16 bits record the actual number of errors. When the original value is directly input into the decimal scalar model, the large radix of the high-order bits (usually in...) The magnitude will completely mask the small error counts at the lower bits (usually in the order of magnitude). Numerical fluctuations (on the order of magnitude) make it impossible for the split points of the tree model to effectively distinguish changes in the underlying error count. Bit-level splitting is performed on the above five attributes, dividing the original value v into higher-order components. With low-order components : ; in, This indicates rounding down. This indicates the modulo operation.

[0060] Each attribute generates two independent new columns, resulting in a total of 10 new columns for the five attributes. This operation separates two types of physical semantics that are mixed in the same numerical space into independent feature dimensions without losing information.

[0061] Furthermore, considering that both the original attribute and the decoupled high and low bit components exhibit an extreme heavy-tailed distribution in their value range, with values ​​spanning up to [0, ... Even higher. In tree-based split gain calculations, maxima dominate the node splitting direction, leading to overfitting of the model to large numerical samples generated by high-pressure read / write operations in normal business scenarios. To compress the numerical space and preserve the relative order, this embodiment applies a logarithmic translation transformation to all 21 original attribute columns and 10 decoupled columns (31 columns in total) in the basic attribute set. Optionally, the specific transformation formula is as follows: ; in, , These represent the SMART attribute values ​​before and after the logarithmic translation transformation, respectively.

[0062] Through logarithmic translation transformation, this embodiment will [0, The interval is smoothly mapped to [0, 13.8], strictly preserving the temporal monotonicity of features and the relative ordering between samples.

[0063] After bit-level decoupling and logarithmic compression, the original data is transformed into a 31-dimensional set of basic features that have undergone semantic restoration and numerical calibration.

[0064] This embodiment further reveals that the core of constructing multi-scale temporal features lies in the effective representation of mutational evolution rates and acceleration trends, rather than simply extending the statistical window. Based on this, this embodiment constructs an evolutionary feature set across three physical scales—short-term mutations, long-term degradation, and nonlinear inflection points—for 12 candidate anomalous attributes in the temporal dimension. Specifically, the multi-scale temporal features constructed for the candidate anomalous attributes in this embodiment include: difference features, slope features, and speedup features. The construction of each type of feature is explained below.

[0065] (1) The transient difference operator is designed for the detection of short-term mutations. For each candidate attribute of anomalies, it extracts... Forward difference of the day . and They represent Time and The attribute value at any given time; As a discrete approximation of the first derivative, it directly reflects the magnitude of newly added abnormal events on a single day; and This approach smooths out sampling jitter introduced by daytime job scheduling while preserving short-term trend information. A total of 36 differential features are generated across 12 attributes and 3 windows.

[0066] (2) To characterize the trend of long-term degradation slope towards chronic wear, a historical observation window of the past 30 days (n=30) was selected with the target time t as the endpoint. For each abnormal candidate attribute, the least squares method was used to fit the linear degradation slope. This window is strictly limited to backward historical data and does not include any observations after t, ensuring that the feature construction follows temporal causal constraints. The average degradation rate of the equipment over a macroscopic timescale was quantified, reflecting the long-term trend of attributes continuously deteriorating from normal levels to the failure threshold. Twelve attributes generated a total of twelve slope features.

[0067] (3) The acceleration ratio feature is used to detect whether the degradation rate suddenly accelerates, which is achieved by comparing the average rate of change within short and long windows. The short-term rate is defined. With long-term rate The acceleration ratio characteristic Accel is calculated as follows: ; in, In this embodiment, the constant is represented. .

[0068] When the acceleration ratio is much greater than 1, it indicates that the recent rate of deterioration has significantly exceeded the historical average, and the equipment may be in a phase of accelerated degradation. The 12 attributes generate a total of 12 columns of acceleration characteristics.

[0069] The daily outbreak indication information corresponding to the 12 candidate anomaly attributes is retained as an auxiliary representation for the 12 columns. After the above processing, a total of 72 columns of derived features are generated for the 12 candidate anomaly attributes.

[0070] Furthermore, considering that difference and division operations can easily generate extreme values ​​in the original numerical space, such as a large difference resulting from a sudden increase in error count from zero to hundreds, or a sharp amplification of the ratio when the long-term rate is close to zero, these will dominate the split gain allocation of the tree model if left untreated. Therefore, this embodiment applies a quantile transformation based on empirical cumulative distribution to all 48 columns of the difference and acceleration columns, forcing the long-tailed distribution to be mapped to a uniform distribution of [0,1]. This suppresses absolute amplitude while preserving the relative magnitude relationships of temporal features. Since the slope feature distribution generated by least squares fitting is relatively stable, and the burst indicator is a binary variable, neither of these two types of features participates in this transformation. After stage-by-stage processing, 31 basic features and 72 derived features together constitute a 103-dimensional sample feature set.

[0071] When training, validating, and testing the model, the sample feature set corresponding to the sample is used as the model input.

[0072] Since multi-scale feature construction generates a large number of candidate features, which may include redundant features, unstable features, and randomly related features, this embodiment further filters the constructed sample feature set.

[0073] Specifically, S4 also includes: during model training and validation, enabling the model to output the contribution of each feature to the prediction result, and updating the sample feature set through sample filtering; sample filtering includes: Features with a missing proportion higher than the preset removal threshold, or features whose values ​​remain unchanged for a duration exceeding the preset duration, are removed. For multiple features with a similarity higher than a preset similarity threshold, only the feature with the highest contribution is retained; Features that contribute more than a preset contribution threshold in the training set but whose contribution is unstable in the validation set (e.g., alternating between contributions above and below the threshold in the validation set) are removed.

[0074] After sample filtering, this embodiment selects the final feature set used for model training. This feature set retains multi-scale temporal change information while reducing the interference of redundant and noisy features on model training.

[0075] In practical applications, tree models, gradient boosting tree models, or other classification models suitable for tabular time-series features can be selected. Optionally, step S4 of this embodiment uses LightGBM as the core classifier to predict the probability of an impending solid-state drive failure.

[0076] During the training phase, this embodiment further adjusts the ratio of positive to negative samples to a range that the classifier can effectively optimize, based on the candidate training set formed by anomaly perception hierarchical sampling. This adjusts the model's focus on fault samples; for example, one fault sample is input for every five healthy samples. This process does not change the label definition of the fault prediction samples, but rather alleviates the model bias caused by an excessive difference in the number of healthy and fault samples through the organization of training samples.

[0077] In practical applications, multiple models can be trained and their output risk probabilities integrated to improve the stability of fault risk ranking.

[0078] In this embodiment, after model training is completed, a fault alarm threshold is determined on the validation set. Since large-scale storage systems have strict limitations on false positive rates, this embodiment does not use a fixed default threshold. Instead, it sets a set of candidate fault alarm thresholds and calculates the device-level false positive rate and fault recall rate for each threshold. When the device-level false positive rate corresponding to a certain threshold does not exceed a preset upper limit, it is included in the candidate deployment threshold set, and the threshold with the better fault recall rate is selected as the final fault alarm threshold. In this way, the risk probability output by the model is transformed into a fault prediction result that meets the low false positive deployment constraint.

[0079] During the offline testing phase, the latest SMART attribute time-series data of the solid-state drive to be predicted is received. Basic feature processing and multi-scale feature construction are completed according to the aforementioned steps. The generated features are then input into the trained fault prediction model to obtain the risk probability of the solid-state drive failing within the prediction lead time. When the risk probability is greater than or equal to the fault alarm threshold, a high-risk alarm is output; when the risk probability is lower than the fault alarm threshold, a normal or low-risk status is output.

[0080] Based on this embodiment, and verified using a publicly available SSD runtime dataset, this invention effectively alleviates the extreme class imbalance and fault feature dilution problems in SSD fault prediction by relying solely on the general SMART attribute. Under the constraint of a device-level false positive rate not exceeding 0.1%, the fault recall rate of this invention reaches 2.27 times the best result among the comparative methods; compared with the representative method in the field, the F1 score is improved by 19.3%. Ablation experiments show that after removing multi-scale temporal features, the overall ranking performance for imbalanced data decreases by 67.2%; after replacing anomaly-aware hierarchical sampling with random undersampling, the fault recall rate under low false positive constraints decreases by 40.9%. Therefore, this invention improves the recall capability and practical deployability of SSD fault prediction under low false positive deployment constraints through anomaly-aware hierarchical sampling, multi-scale temporal feature enhancement, and low false positive threshold optimization, providing early warning basis for data migration, equipment replacement, and risk management.

[0081] Overall, this embodiment, relying solely on the general SMART attribute, collaboratively optimizes aspects such as data distribution, feature construction, model training, and deployment decisions. It effectively mitigates the decision boundary shift caused by extreme class imbalance, avoids the smoothing out of short-term mutation signals by long-window statistics, and adapts to the requirements for alarm threshold selection and result interpretation under low false alarm rate constraints.

[0082] Example 2: A solid-state drive (SSD) fault prediction method based on anomaly perception hierarchical sampling includes: inputting the SMART attribute of the SSD to be predicted into the SSD fault prediction model to obtain the probability of the SSD failing within a preset fault warning window before the prediction time, comparing it with a fault alarm threshold, and if it is greater than or equal to the fault alarm threshold, then predicting that the SSD to be predicted will fail soon. The solid-state drive (SSD) fault prediction model is established using the SSD fault prediction model establishment method based on anomaly perception hierarchical sampling provided in Example 1 above.

[0083] When the probability output by the fault prediction model is lower than the fault alarm threshold, it outputs a normal or low-risk status.

[0084] This embodiment corresponds to the online prediction stage of the established SSD fault prediction model. For SSDs that trigger high-risk alarms, this embodiment records the device identifier, sampling time, fault risk probability, alarm threshold, and main anomaly SMART attributes, which are then used by the subsequent operation and maintenance system to perform data migration, device replacement, or risk handling.

[0085] To enhance the interpretability of the prediction results, this embodiment generates instance-level feature attribution results for the SSD that triggered the alarm when predicting an impending SSD failure. Specifically, it calculates the positive or negative contribution of each feature to the current prediction result, which can be obtained through SHAP attribution analysis. Based on this, it can further output SMART attributes whose contributions exceed the relevant thresholds and their corresponding changes, enabling operations and maintenance personnel to understand which attribute anomalies, degradation trends, or accelerating trends are primarily driving this alarm.

[0086] The feature attribution results output in this embodiment serve as supplementary explanatory information for risk warnings and do not change the model prediction results.

[0087] Example 3: A computer-readable storage medium includes a stored computer program; when the computer program is executed by a processor, it implements the solid-state drive fault prediction model establishment method based on anomaly perception hierarchical sampling provided in Embodiment 1 above, and / or the solid-state drive fault prediction method based on anomaly perception hierarchical sampling provided in Embodiment 2 above.

[0088] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for establishing a solid-state drive (SSD) fault prediction model based on anomaly perception and hierarchical sampling, characterized in that, include: S1: Collect SMART attribute timing data of each solid-state drive according to the preset sampling period, and record the status of the solid-state drive at each sampling time. Mark the SMART attribute record within the preset fault warning window before the fault time as a fault sample, and mark the SMART attribute record before the fault warning window as a healthy sample. S2: Count the number of SMART attributes that show abnormal changes in each healthy sample relative to the previous sampling time. This number is used as the short-term abnormal concurrency of the healthy samples. The healthy sample set is then divided into groups with a short-term abnormal concurrency greater than [a certain value]. The adversarial hard negative sample set and short-term abnormal concurrency belong to A high-risk healthy sample set and a short-term abnormal concurrency level of less than A low-risk healthy sample set; ; S3: According to the first retention ratio respectively Second retention ratio The high-risk health sample set and the low-risk health sample set are sampled, and the sampled health samples, adversarial hard negative sample set, and fault samples, together with the corresponding state labels, constitute the prediction sample set. ; S4: Divide the prediction sample set into a training set, a validation set, and a test set according to a preset ratio. Use the training set to train a machine learning model to obtain a solid-state drive failure prediction model for predicting the probability of an impending solid-state drive failure based on the SMART attribute. Then, use the validation set and the test set to validate and test the model, thus completing the establishment of the solid-state drive failure prediction model.

2. The method for establishing a solid-state drive fault prediction model based on anomaly perception hierarchical sampling as described in claim 1, characterized in that, Between S3 and S4, there is also: Remove the SMART attributes corresponding to the manufacturer's normalized score from each sample, and select the SMART attributes that are most correlated with the health status of the solid-state drive from the remaining SMART attributes to obtain the basic attribute set; perform basic feature processing on the SMART attributes in the basic attribute set to obtain the sample feature set corresponding to each sample. Furthermore, in S4, when training, validating, and testing the solid-state drive failure prediction model using the training set, validation set, and test set, the sample feature set corresponding to each sample is used as the model input. The basic feature processing includes: Each SMART attribute is treated as a feature and added to the sample feature set; According to semantics, the SMART attribute, which uses composite coding logic, is split into multiple sub-features, and the resulting sub-features are also added to the sample feature set. Apply a logarithmic translation transformation to each feature in the sample feature set.

3. The method for establishing a solid-state drive fault prediction model based on anomaly perception hierarchical sampling as described in claim 2, characterized in that, The set of basic attributes includes abnormal candidate attributes that will undergo sudden changes within a preset fault warning window before the fault occurs. Furthermore, the following is included between S3 and S4: Multi-scale temporal features are constructed for each anomaly candidate attribute in the basic attribute set, and then added to the sample feature set; The multi-scale temporal features include: difference features; The differential features are constructed in the following ways: Calculate the target sampling time attribute values Compared with its predecessor Each sampling time attribute values The result of the difference operation ; For the result of the difference operation By applying a quantile transformation based on the empirical cumulative distribution, the difference features are obtained; in, .

4. The method for establishing a solid-state drive fault prediction model based on anomaly perception hierarchical sampling as described in claim 3, characterized in that, The multi-scale temporal features also include: slope features; The slope feature is constructed in the following ways: At the target sampling time The attribute values ​​within the previously preset historical observation window were fitted with a linear degradation slope using the least squares method. With linear degradation slope As a slope feature.

5. The method for establishing a solid-state drive fault prediction model based on anomaly perception hierarchical sampling as described in claim 4, characterized in that, The multi-scale temporal features also include: speedup features; The construction methods of the speedup ratio feature include: Calculate short-term rate and long-term rate and in accordance with Calculation speedup ; For acceleration ratio Applying a quantile transformation based on the empirical cumulative distribution yields the speedup characteristics; in, The length of the historical observation window, For the target sampling time attribute values Compared with its predecessor Each sampling time attribute values The result of the difference operation; This is a preset constant.

6. The method for establishing a solid-state drive fault prediction model based on anomaly perception hierarchical sampling as described in claim 5, characterized in that, S4 further includes: during model training and validation, enabling the model to output the contribution of each feature to the prediction result, and updating the sample feature set through sample filtering; the sample filtering includes: Features with a missing proportion higher than the preset removal threshold, or features whose values ​​remain unchanged for a duration exceeding the preset duration, are removed. For multiple features with a similarity higher than a preset similarity threshold, only the feature with the highest contribution is retained; Features whose contribution to the training set is higher than the preset contribution threshold, but whose contribution to the validation set is unstable, are removed.

7. The method for establishing a solid-state drive fault prediction model based on anomaly perception hierarchical sampling as described in any one of claims 1 to 6, characterized in that, S4 also includes: setting a set of candidate fault alarm thresholds, wherein the fault alarm thresholds are used to determine whether the solid-state drive is about to fail; when the probability output by the solid-state drive prediction model is greater than or equal to the fault alarm threshold, it is determined that the solid-state drive is about to fail. For each fault alarm threshold in the candidate fault alarm threshold set, the device-level false alarm rate and fault recall rate on the verification set are calculated respectively, and the fault alarm threshold with the corresponding device-level false alarm rate not exceeding the preset upper limit is added to the candidate deployment threshold set. The fault alarm threshold with the optimal fault recall rate is selected from the candidate deployment threshold set as the final fault alarm threshold.

8. A solid-state drive fault prediction method based on anomaly perception hierarchical sampling, characterized in that, include: Input the SMART attribute of the solid-state drive to be predicted into the solid-state drive failure prediction model to obtain the probability of the solid-state drive failing within the preset failure warning window before the prediction time. Compare it with the failure alarm threshold. If it is greater than or equal to the failure alarm threshold, it is predicted that the solid-state drive to be predicted will fail soon. The solid-state drive fault prediction model is established by the solid-state drive fault prediction model establishment method based on anomaly perception hierarchical sampling as described in any one of claims 1 to 7.

9. The solid-state drive fault prediction method based on anomaly perception hierarchical sampling as described in claim 8, characterized in that, Also includes: When predicting that a solid-state drive (SSD) is about to fail, feature attribution results are generated. The feature attribution results include: the positive or negative contribution of each feature to the current prediction result.

10. A computer-readable storage medium, characterized in that, Includes a stored computer program; when the computer program is executed by a processor, it implements the solid-state drive fault prediction model establishment method based on anomaly perception hierarchical sampling as described in any one of claims 1 to 7, and / or the solid-state drive fault prediction method based on anomaly perception hierarchical sampling as described in claim 8 or 9.

Images