Disk anomaly detection method and device, medium, equipment and product
By constructing a histogram probability density model and a comprehensive evaluation model, and combining the disk hardware status and business dependency characteristics, the problem of low accuracy in disk anomaly detection in existing technologies is solved, and accurate prediction and efficient detection of disk failures are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-25
- Publication Date
- 2026-04-07
AI Technical Summary
Existing disk anomaly detection methods fail to effectively combine business scenarios and have low accuracy in fault prediction, making it difficult to accurately reflect the health status of disks under real business loads.
Hardware anomaly detection is achieved by constructing a histogram probability density model and combining it with a comprehensive evaluation model of disk service performance indicators, while also incorporating application component dependency characteristics, thus enabling dual judgment at both the hardware and service levels.
It improves the accuracy and reliability of disk failure prediction, accurately matches the actual business scenario needs of resource pools/data centers, reduces false alarm rates, and ensures the real-time and efficient nature of detection.
Smart Images
Figure CN121807634A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of detection technology, and in particular to a disk anomaly detection method, apparatus, computer-readable storage medium, electronic device, and computer program product. Background Technology
[0002] With the rapid development of big data and cloud computing technologies, the daily data volume generated by resource pools and data centers is exploding, and most of this data is stored on disks. As the core storage medium, disk failure can lead to component replacement, business downtime, and even data loss, significantly impacting the stable operation of businesses. Therefore, it is crucial to detect and replace disks in advance to mitigate their negative impact on business systems. Summary of the Invention
[0003] The purpose of this invention is to provide a disk anomaly detection method, apparatus, computer-readable storage medium, electronic device, and computer program product that performs a series verification from hardware anomaly detection of the target disk to business health detection. This method considers both the disk's own hardware status and the application component dependency characteristics, making the disk's business health status accurately match the actual business scenario requirements of the resource pool / data center. This can effectively improve the accuracy and reliability of disk fault prediction.
[0004] A first aspect of the present invention provides a disk anomaly detection method, comprising: Based on the first hardware parameter feature set of the target disk in the first time period, a histogram probability density model is constructed, and hardware anomaly detection is performed on the second hardware parameter feature set in the second time period through the histogram probability density model. When the hardware anomaly detection result indicates that the target disk is a suspected abnormal disk, the first comprehensive component disk dependency set and the first hardware parameter feature set of the suspected abnormal disk in the first time period are input into the comprehensive evaluation model of disk business performance indicators to obtain the first comprehensive evaluation set of business performance indicators, so as to determine the first anomaly threshold. The second comprehensive component disk dependency set and the second hardware parameter feature set of the suspected abnormal disk in the second time period are input into the comprehensive evaluation model to obtain the second comprehensive evaluation set of business performance indicators and calculate the detection mean. The business health status of suspected abnormal disks is obtained by comparing the first abnormal threshold with the detection mean.
[0005] Optionally, the method further includes: When the hardware anomaly detection result indicates that the target disk is not a suspected anomaly disk, the detection of business health status is stopped, and the hardware anomaly detection result is output directly.
[0006] Optionally, the feature items included in the first hardware parameter feature set and the second hardware parameter feature set are composed of representative comprehensive indicators and weakly correlated hardware parameter indicators; wherein, the comprehensive indicators are obtained by principal component analysis to reduce the dimensionality of highly correlated hardware parameter indicators.
[0007] Optionally, the step of performing hardware anomaly detection on the second hardware parameter feature set within the second time period using the histogram probability density model includes: Based on the histogram probability density model, the first anomaly detection score set corresponding to the first hardware parameter feature set and the second anomaly detection score set corresponding to the second hardware parameter feature set are obtained respectively. Based on the first anomaly detection score set, the first outlier mean and the corresponding outlier difference are calculated to determine the second anomaly threshold. The second outlier mean was calculated based on the second anomaly detection score set. The hardware anomaly detection result is obtained by comparing the second outlier mean and the second anomaly threshold.
[0008] Optionally, both the first integrated component disk dependency set and the second integrated component disk dependency set are calculated using a disk I / O dependency model; wherein, the disk I / O dependency model is constructed through the following steps: Obtain the historical sample set of the first indicator item when each application component runs independently, and train the disk I / O dependency sub-model of the corresponding application component based on each historical sample set; When application components run together, the collected data of each application component regarding the first indicator item are input into the corresponding disk I / O dependency sub-model. The output results of each disk I / O dependency sub-model are weighted to obtain the disk I / O dependency model.
[0009] Optionally, the weights corresponding to the output results are determined by the workload proportion of the corresponding application components, or obtained through end-to-end model learning.
[0010] Optionally, the first indicator includes: corresponding component performance indicators, server performance indicators, and disk performance indicators.
[0011] Optionally, the comprehensive evaluation model for disk service performance indicators is constructed through the following steps: Obtain a sample set of business performance metrics for the target disk and assign weights to it to obtain a set of disk business performance evaluation labels after merging the metrics; Based on the disk service performance evaluation tag set and the corresponding third comprehensive component disk dependency set, obtain the service health status evaluation tag set of the target disk; Based on the business health status evaluation label set and the corresponding third hardware parameter feature set, a multivariate linear regression fitting is performed to obtain a comprehensive evaluation model for the disk business performance indicators.
[0012] Optionally, when merging indicators, the weight of each indicator to be weighted is determined by the dispersion of the individual indicator data and the total dispersion of all indicator data.
[0013] Optionally, the first time period is updated on a rolling basis according to a preset time cycle; wherein the first time period is earlier than the second time period.
[0014] Optionally, comparing the first anomaly threshold and the detection mean to obtain the service health status of the suspected abnormal disk includes: When the average detection value is not greater than the first abnormal threshold, the business health status is a non-abnormal status; When the average detection value is greater than the first abnormal threshold, the business health status is abnormal, and the suspected abnormal disk is determined to have an anomaly.
[0015] A second aspect of the present invention provides a disk anomaly detection device, comprising: The hardware anomaly detection module is used to construct a histogram probability density model based on the first hardware parameter feature set of the target disk in the first time period, and to perform hardware anomaly detection on the second hardware parameter feature set in the second time period through the histogram probability density model. The anomaly threshold acquisition module is used to input the first comprehensive component disk dependency set and the first hardware parameter feature set of the suspected abnormal disk within the first time period into the comprehensive evaluation model of disk business performance indicators when the hardware anomaly detection result indicates that the target disk is a suspected abnormal disk, so as to obtain the first comprehensive evaluation set of business performance indicators and determine the first anomaly threshold. The detection mean acquisition module is used to input the second comprehensive component disk dependency set and the second hardware parameter feature set of the suspected abnormal disks in the second time period into the comprehensive evaluation model to obtain the second comprehensive evaluation set of business performance indicators and calculate the detection mean. The health status acquisition module is used to obtain the business health status of suspected abnormal disks by comparing the first abnormal threshold with the detection average.
[0016] A third aspect of the present invention provides a computer-readable storage medium comprising a stored computer program; wherein, when the computer program is executed, it controls the device in which the computer-readable storage medium is located to perform the disk anomaly detection method described in any embodiment of the first aspect.
[0017] A fourth aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the disk anomaly detection method described in any embodiment of the first aspect.
[0018] A fifth aspect of the present invention provides an electronic device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the disk anomaly detection method according to any embodiment of the first aspect.
[0019] Compared with existing technologies, this invention provides a disk anomaly detection method, apparatus, computer-readable storage medium, electronic device, and computer program product. By constructing a histogram probability density model, hardware anomaly detection is performed on a second set of hardware parameter features within a second time period to achieve initial screening of target disk anomalies at the hardware level. Then, business health status verification is initiated for suspected abnormal disks, constructing a dual judgment mechanism of "hardware detection - business verification." This leverages the early warning advantage of hardware parameter detection while effectively avoiding false alarms caused by a single detection dimension through business layer indicator verification, significantly improving the accuracy and reliability of fault diagnosis. In the business health status verification stage, a comprehensive component disk dependency is introduced, linking the disk status with the actual performance of upper-layer services, directly reflecting the degree of impact on the business system. This ensures that the disk's business health status accurately matches the actual business scenario requirements of the resource pool / data center, thereby effectively improving the accuracy and reliability of disk fault prediction. Attached Figure Description
[0020] Figure 1 This is a flowchart illustrating an embodiment of the disk anomaly detection method provided by the present invention; Figure 2 This is a flowchart illustrating another embodiment of the disk anomaly detection method provided by the present invention; Figure 3 This is a schematic diagram of the structure of an embodiment of the disk anomaly detection device provided by the present invention; Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device provided by the present invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] The core data foundation for disk failure prediction is the disk's hardware parameters, specifically the SMART (Self-Monitoring Analysis and Reporting Technology) metric set. This is a technical standard for automatically monitoring the health of disk drives and reporting potential problems. As an industry-wide standard, SMART specifies various standards that disk manufacturers should follow. The SMART specification metric set provides 255 standard metrics, and disk manufacturers can support all or some of these metrics as needed.
[0023] During their research, the inventors discovered that the operating status of a disk is closely related to the characteristics of the programs and services hosted on the server. Different programs and services have significantly different degrees of dependence on CPU, memory, and I / O resources, and these differences directly affect the actual operating load and failure risk of the disk.
[0024] Specifically, compute-intensive components (such as Nginx handling high-concurrency requests or PostgreSQL executing complex queries) consume significant CPU resources; the performance of in-memory databases (such as Redis) is highly dependent on memory configuration, as their data operations are performed directly in memory; and data-intensive components (such as MySQL in high-concurrency environments) frequently consume large amounts of I / O resources. Therefore, even if two disks have identical hardware configurations, in actual operation, the disk hosting the MySQL service often exhibits a higher probability of failure than the disk hosting the Redis service, and its service capacity is also relatively lower.
[0025] Currently, there are two main methods for disk anomaly detection: single-threshold detection methods based on the SMART metric and anomaly detection methods based on machine learning. During the implementation of these disk anomaly detection methods, the inventors discovered the following drawbacks in both methods: (1) Single threshold detection method based on SMART index: The disk status is monitored by setting a fixed threshold for the SMART index. Although it is simple to implement, the judgment basis is relatively one-sided, the early warning accuracy is low, it is difficult to conduct a comprehensive analysis of the disk status, and it cannot effectively achieve early prediction of faults. (2) Anomaly detection methods based on machine learning: Models are built using algorithms such as k-NN, LOF, CNN, and GAN, and are trained and validated using a large number of negative disk failure samples. However, collecting negative failure samples is time-consuming and labor-intensive, and labeling is difficult. Moreover, the collected disk failure information is often disk samples that cannot work directly after a hardware failure. However, the definition of disk failure in actual resource pools or data centers is defined from the perspective of the degree of impact of the disk on the business system. That is to say, even if the disk can still work, it can be defined as a problem with the health status of the disk business based on the degree of impact on the operation of the business system (such as disk IO waiting time, disk IO utilization rate, etc.). From the perspective of operation and maintenance, the disk needs to be maintained or replaced. Existing public datasets (such as the Backblaze dataset) have extremely low compatibility with the actual resource pools / data centers of target companies in terms of disk brands and models. Due to technical confidentiality, there are differences in the SMART metric outputs of disks from different manufacturers and models, resulting in poor training performance of models in cross-manufacturer and cross-model scenarios. This leads to low prediction accuracy of machine learning-based anomaly detection methods, which mostly remain in the laboratory research stage and are difficult to implement in practice.
[0026] Furthermore, neither of the above two methods takes into account the differences in the characteristics of the programs carried by the disk. They only rely on the disk's own hardware parameter indicators (SMART) for detection, which leads to bias in the prediction results and makes it difficult to accurately reflect the health status of the disk under real business load (i.e., disk business health status).
[0027] Therefore, in order to address the problem that the above-mentioned disk anomaly detection methods do not take into account business scenarios and have low accuracy in fault prediction, this invention provides a disk anomaly detection method, device, computer-readable storage medium, electronic device, and computer program product. By comprehensively considering the disk hardware status and business dependency characteristics, it effectively improves the accuracy and reliability of disk fault prediction.
[0028] See Figure 1 This is a flowchart illustrating an embodiment of the disk anomaly detection method provided by the present invention.
[0029] A first aspect of the present invention provides a disk anomaly detection method, including steps S1 to S4, as follows: Step S1: Based on the first hardware parameter feature set of the target disk in the first time period, construct a histogram probability density model, and use the histogram probability density model to detect hardware anomalies in the second hardware parameter feature set in the second time period.
[0030] It should be noted that the first time period is the historical training period, which can be set to the most recent 90 days of the target disk. SMART raw indicator data of the target disk is collected every 5 minutes, accumulating to 288 × 90 = 25920 sample data points over 90 days, constituting the raw hardware parameter indicator set. The original hardware parameter index set is subjected to feature engineering to obtain the first hardware parameter feature set after removing redundant indices. It can comprehensively reflect the hardware health status of the disk in the smallest dimensions.
[0031] The Histogram-based Incremental Outlier Detection (HIOD) method is employed, utilizing the first hardware parameter feature set. Construct a histogram probability density model and calculate the corresponding HIOD anomaly detection score set (i.e., the first anomaly detection score set). Among them, the histogram probability density model is mainly used to characterize the distribution characteristics of hardware parameters of the disk under non-abnormal conditions.
[0032] This invention employs a Flink-based big data streaming parallel computing architecture. Therefore, the second time period is the current detection cycle, which can be set to the most recent 60 minutes of the target disk. Similarly, the raw SMART index data of the target disk is acquired at a 5-minute acquisition cycle, and after undergoing the same feature engineering processing, a second hardware parameter feature set is obtained. (i.e., online anomaly detection dataset), which can avoid misjudgment caused by a single collection of anomalies, thereby improving the stability of prediction results.
[0033] The second hardware parameter feature set The input is fed into the constructed histogram probability density model to calculate the corresponding HIOD anomaly detection score set (i.e., the second anomaly detection score set). ).
[0034] Finally, through the first anomaly detection score set Second Anomaly Detection Score Set By combining the three-standard-deviation (3σ) principle, hardware anomaly detection of the current target disk can be achieved.
[0035] Step S2: When the hardware anomaly detection result indicates that the target disk is a suspected abnormal disk, the first comprehensive component disk dependency set and the first hardware parameter feature set of the suspected abnormal disk in the first time period are input into the comprehensive evaluation model of disk business performance indicators to obtain the first comprehensive evaluation set of business performance indicators, so as to determine the first anomaly threshold.
[0036] It should be noted that if the target disk is suspected of being abnormal, a disk service health check will be triggered to perform closed-loop detection and verification; otherwise, no further detection is required.
[0037] The comprehensive evaluation model for disk performance metrics is a multiple linear regression model. Its inputs include a hardware parameter feature set and the disk dependency of integrated components, and its output is a comprehensive evaluation value of the performance metrics. The integrated component disk dependency is calculated based on the weighted average of the dependencies of various application components (such as MySQL, Kafka, Elasticsearch, Zookeeper, Redis, and RabbitMQ) on disk I / O resources hosted on the target disk. The first integrated component disk dependency set and the first hardware parameter feature set within the first time period are input into the comprehensive evaluation model to obtain the first comprehensive evaluation set of the performance metrics. Combined with the 3σ principle, the first anomaly threshold (i.e., the critical value of the target disk's performance under normal hardware and business load) is obtained.
[0038] Step S3: Input the second comprehensive component disk dependency set and the second hardware parameter feature set of the suspected abnormal disk in the second time period into the comprehensive evaluation model to obtain the second comprehensive evaluation set of business performance indicators and calculate the detection mean.
[0039] It should be noted that the second set of disk dependencies of the second integrated component and the second set of hardware parameter features within the second time period are input into the same comprehensive evaluation model to obtain the second comprehensive evaluation set of business performance indicators. The root mean square method is used to calculate the detection mean of the second comprehensive evaluation set, which is used to characterize the overall level of disk business performance within the current detection period.
[0040] Step S4: By comparing the first anomaly threshold with the detection mean, the business health status of the suspected abnormal disk is obtained.
[0041] It should be noted that, in this embodiment of the invention, the business health status of the suspected abnormal disk is finally determined by comparing the real-time business performance level (i.e., the detection mean) with the benchmark threshold (i.e., the first abnormal threshold) and combining the business performance abnormality verification logic, thus forming a closed-loop detection.
[0042] As can be seen from the above, the disk anomaly detection method provided by the embodiments of the present invention, from the hardware anomaly detection of the target disk to the serial verification (closed-loop judgment mechanism) of business health detection (business performance anomaly detection), takes into account both the hardware status of the disk itself and the dependency characteristics of application components, so that the business health status of the disk accurately matches the actual business scenario requirements of the resource pool / data center, and can effectively improve the accuracy and reliability of disk failure prediction.
[0043] In an optional embodiment, the method further includes: When the hardware anomaly detection result indicates that the target disk is not a suspected anomaly disk, the detection of business health status is stopped, and the hardware anomaly detection result is output directly.
[0044] It should be noted that, in this embodiment of the invention, after performing hardware anomaly detection on the second hardware parameter feature set based on the histogram probability density model in step S1, if the hardware anomaly detection result indicates that the target disk is a suspected abnormal disk, then the business health status detection (steps S2 to S4) is initiated for serial verification; if the hardware anomaly detection result indicates that the target disk is not a suspected abnormal disk, then the triggering is stopped and the business health status detection is skipped, and its hardware anomaly detection result is directly adopted.
[0045] By setting up a pre-judgment branch, this invention can effectively filter out most disks that are in a healthy state, eliminating the need for them to enter the more complex business performance comprehensive evaluation stage. This significantly reduces the overall computing load of the system, making it particularly suitable for online monitoring scenarios of massive disks in resource pools / data centers, ensuring the real-time performance and efficiency of the detection process.
[0046] In an optional embodiment, the first time period is updated on a rolling basis according to a preset time period; wherein, the first time period is earlier than the second time period.
[0047] It should be noted that, in this embodiment of the invention, the first time period is updated on a rolling basis with a preset time period of days, which can automatically trigger the update process of the dataset / model.
[0048] For example, if the first time period is defined as "the most recent 90 days", then at each update cycle point, the system will automatically slide the data window (90 days in length) forward by 1 day. This process is usually completed efficiently based on time series databases such as ClickHouse.
[0049] Under this rolling update mechanism, the first time period used for model training is always earlier than the second time period used for real-time detection on the timeline. As the data from the first time period is updated, the histogram probability density model and the comprehensive evaluation model based on it, along with disk business performance indicators, will also be periodically retrained and have their parameters iterated. This allows the model to perfectly match the health status of the disk at its current lifespan, maintaining the model's "freshness" and predictive accuracy.
[0050] In an optional embodiment, the features included in the first hardware parameter feature set and the second hardware parameter feature set are both composed of representative comprehensive indicators and weakly correlated hardware parameter indicators; wherein, the comprehensive indicators are obtained by principal component analysis to reduce the dimensionality of highly correlated hardware parameter indicators.
[0051] It should be noted that the first and second hardware parameter feature sets used for model training and anomaly detection in this embodiment of the invention do not directly use all the originally collected SMART metrics, but are feature sets with high representativeness and low redundancy, which are composed of two types of metrics: (1) Representative comprehensive indicators: First, select hardware parameter indicators that are highly correlated with each other from the original indicators to form a set of related indicators; then apply principal component analysis to the set of related indicators to perform feature dimensionality reduction, and retain the top principal components with a cumulative contribution rate of more than 85% as comprehensive indicators.
[0052] (2) Weakly correlated hardware parameter indicators: In the original SMART indicators, indicators that are weakly correlated with other indicators but have independent indicative significance for the health status of disk hardware are not included in the feature reconstruction of principal component analysis, but are directly retained.
[0053] Specifically, the original disk SMART metric set is a high-dimensional dataset with 255 dimensions. Different metrics have different correlations and influences on disk health status. This embodiment of the invention uses a data dimensionality reduction method to process the original metric data and define a set of feature items for disk hardware SMART metrics.
[0054] The first step is to divide the indicators into two subsets based on their statistical correlation: a set of highly correlated indicators. and weak correlation index set .
[0055] Calculate the covariance between all pairwise SMART indicators to form a covariance matrix. If there are 255 SMART indicators, the size of the covariance matrix will be 255×255. This covariance matrix can visually represent the correlation distribution among all indicators, and indicators with an absolute covariance value greater than 1 are grouped into a highly correlated indicator set. The remaining indicators are classified into the weakly correlated indicator set. .
[0056] For example, ={Disk data read / write error rate, write error rate, read error block count, seek error rate, seek performance, programming failure block count, erase / write failure block count, wear block count...}; ={Number of remapped sectors, disk power-on technology, temperature, vibration...}.
[0057] The second step is to develop a set of highly correlated indicators. There is a problem of redundant indicators in the data, which requires non-standardized dimensionality reduction through principal component analysis (PCA) to generate fewer, more concentrated and independent comprehensive indicators.
[0058] For highly correlated indicator sets The n collected samples are standardized to eliminate the influence of dimensions, solve the comparability problem between data indicators, and make the data of the same order of magnitude, resulting in a standardization matrix Z; the specific standardization formula is: ; In this formula, The value of the i-th index of the j-th sample in the original sample; It is the standardized value of the i-th indicator of the j-th sample; n is the number of samples.
[0059] Based on the standardized matrix Z, the correlation coefficient matrix R between the indicators is calculated using the following formula: ; In this formula, It is the correlation coefficient between the i-th indicator and the j-th indicator; It is the value of the i-th index of the k-th sample in the standardized matrix Z; It is the value of the j-th indicator of the k-th sample in the standardized matrix Z; n is the number of samples; m is the set of highly correlated indicators. The number of indicators.
[0060] By performing eigenvalue decomposition on the correlation coefficient matrix R, the eigenvalues can be obtained. and the corresponding feature vector (Contains m values). The first... One principal component variable It is a linear combination of the original correlation indicators, expressed as: In this formula, It is an eigenvector The first in A number; It is a set of highly correlated indicators The first in One original indicator.
[0061] Calculate each principal component The corresponding cumulative contribution rate is used to retain the principal components with a cumulative contribution rate greater than 85% as the final representative comprehensive indicator.
[0062] The third step is to combine the comprehensive index obtained through PCA dimensionality reduction with the initially defined set of weakly correlated indicators. The features are merged to form the SMART indicator feature set. It is used for model training and anomaly detection. By removing redundant indicators, the amount of computation can be significantly reduced, the computation speed can be accelerated, and the overfitting of the model can be avoided, thus achieving the effect of comprehensively reflecting the health status of disk hardware with the fewest possible indicators.
[0063] It is worth noting that the first hardware parameter feature set With the second hardware parameter feature set All of these originate from this. Feature space.
[0064] See Figure 2 This is a flowchart illustrating another embodiment of the disk anomaly detection method provided by the present invention.
[0065] This invention is based on the actual business scenario of disk monitoring and maintenance in resource pools / data centers. From the perspective of the impact of disks on the operation of business systems, it defines a "disk business health status." Disk business performance indicators (such as disk IO utilization and IO wait time) can directly reflect the degree of impact of the current disk on the operation of the business system. However, the factors leading to abnormal disk business performance are diverse, including not only disk hardware malfunctions but also abnormalities in the surrounding environment, abnormal host load, and other host hardware malfunctions. Detecting abnormal disk business performance indicators alone is insufficient to accurately attribute them to abnormal disk hardware parameters, as other factors can also cause these abnormalities.
[0066] To establish a more accurate evaluation system, this embodiment of the invention selects six typical application components—MySQL, Kafka, Elasticsearch, Zookeeper, Redis, and RabbitMQ—as examples. Three types of key metrics are collected during the runtime of these application components: component performance metrics (such as read latency and write latency), server performance metrics (such as CPU utilization and memory utilization), and disk performance metrics (such as IO utilization and IO wait time). Based on these metrics, a disk IO dependency sub-model is constructed for each application component, thereby quantifying the degree of dependence of different components on disk IO resources.
[0067] Based on this, such as Figure 2 As shown, this embodiment of the invention performs correlation analysis on disk service performance indicators, disk I / O dependency of application components, and disk hardware parameter SMART indicators to construct two key models based on machine learning for each disk: a correlation analysis model and an anomaly detection model. The correlation analysis model can be used to calculate the comprehensive evaluation value of disk dependency of integrated components and disk service performance indicators; the anomaly detection model includes hardware anomaly detection and disk service health detection.
[0068] Compared to a single physical disk status assessment, this invention introduces component I / O dependency and business health status, making the anomaly detection results more closely reflect the actual business operation status. This significantly improves the sensitivity, accuracy, and reliability of fault prediction, and is very likely to become the "gold standard" for judging disk health status in actual operation and maintenance scenarios.
[0069] like Figure 2 As shown, the overall structure of this invention embodiment is as follows: (1) Data acquisition module: Real-time acquisition is performed in 5-minute intervals to obtain disk business performance indicators and disk hardware parameter SMART indicators; among them, disk business performance indicators include total disk IO utilization, IO await (disk IO wait time) and disk IO utilization of different application components (based on the actual running components).
[0070] Of course, we will also specifically collect component performance metrics (such as read latency and write latency), server performance metrics (such as CPU utilization and memory utilization), and disk performance metrics for six application component clusters: MySQL, Kafka, Elasticsearch, Zookeeper, Redis, and RabbitMQ. These metrics are used to train calculations on the disk I / O dependency of different application components. Specific metrics collected include: MySQL: Average SQL latency, host CPU / Memory utilization, IO utilization, and IO await; Kafka: Average message publish / subscribe latency, host CPU / Memory utilization, IO utilization, and IO await; ES: query latency, host CPU / Memory utilization, IO utilization, and IO await; ZK: Data read / write latency, host CPU / Memory utilization, IO utilization, and IO await; Redis: query latency, host CPU / Memory utilization, IO utilization, and IO await; RabbitMQ: Average message publish / subscribe latency, host CPU / Memory utilization, IO utilization, and IOawait.
[0071] The metric data collected by the above application components in the cluster are not displayed. Figure 2 middle.
[0072] (2) Data storage and scheduling module: The collected data is stored in a data warehouse on the disk (such as ClickHouse) as historical data storage; on the other hand, it is sent to the online anomaly detection module based on the Flink big data computing framework for online intelligent detection and analysis in a streaming manner.
[0073] (3) Model training module: Obtain data from the data warehouse and construct a correlation analysis model of disk business performance indicators, disk IO dependency of application components and hardware parameter SMART indicators (i.e., a comprehensive evaluation model of disk business performance indicators), and construct a disk anomaly detection model (such as obtaining the first anomaly threshold for disk business health detection and the second anomaly threshold for hardware anomaly detection).
[0074] (4) Online anomaly detection module: Based on the loaded model (association analysis model and anomaly detection model), it performs detection analysis and outputs results, ultimately realizing disk failure prediction.
[0075] Based on the above architecture, the following will describe the process of obtaining the comprehensive evaluation model of the integrated component disk dependency set and disk service performance indicators.
[0076] In an optional embodiment, both the first integrated component disk dependency set and the second integrated component disk dependency set are calculated using a disk I / O dependency model; wherein, the disk I / O dependency model is constructed through the following steps: Obtain the historical sample set of the first indicator item when each application component runs independently, and train the disk I / O dependency sub-model of the corresponding application component based on each historical sample set; When application components run together, the collected data of each application component regarding the first indicator item are input into the corresponding disk I / O dependency sub-model. The output results of each disk I / O dependency sub-model are weighted to obtain the disk I / O dependency model.
[0077] Furthermore, the first indicator includes: the corresponding component performance indicators, server performance indicators, and disk performance indicators.
[0078] It should be noted that both the first comprehensive component disk dependency set (corresponding to the first time period / historical training cycle) and the second comprehensive component disk dependency set (corresponding to the second time period / current detection cycle) are calculated using a unified disk I / O dependency model. The specific construction process is as follows: The first step is to obtain historical sample sets for each of the six application components—MySQL, Kafka, Elasticsearch, Zookeeper, Redis, and RabbitMQ—when running independently (i.e., deployed in a dedicated component cluster, ensuring that only that single component runs on the server disk). Each component's historical sample set contains a first metric consisting of the corresponding component performance metrics, server performance metrics, and disk performance metrics.
[0079] For example, MySQL's historical sample set This includes the CPU utilization of the MySQL server. Memory usage Average SQL execution time in MySQL Server disk I / O utilization Disk I / O wait time Therefore, the historical sample set of MySQL Each sample is formatted as follows: The historical sample set format for other components follows the same structure as described above, but is replaced with the corresponding indicator items for each component.
[0080] The second step involves training a dedicated disk I / O dependency sub-model for each application component using a CNN (Convolutional Neural Network) algorithm, based on historical sample sets from independent operation of each component. This is used to output the disk I / O dependency of the corresponding component.
[0081] For example, for MySQL components, their historical sample sets are used. Training yields a disk I / O dependency sub-model This model can be used to calculate MySQL's disk I / O dependency. : ;in, It is the output of the sub-model, representing the degree of MySQL component dependence on disk I / O under given input metrics.
[0082] Similarly, following the steps above, the disk I / O dependency sub-models for other components can be trained and obtained, and the corresponding disk I / O dependencies can be calculated. That is, the disk I / O dependency of the Kafka component is... The disk I / O dependency of the Elasticsearch component is The disk I / O dependency of the Zookeeper component is The disk I / O dependency of the Redis component is The disk I / O dependency of the RabbitMQ component is .
[0083] In real-world business environments, disks typically run multiple application components simultaneously. The following steps are required to perform fusion calculations and obtain the overall component disk dependencies: During the joint runtime of application components, data on the first metric is first collected in real time from each application component. Then, the real-time collected data from each component is input into its corresponding disk I / O dependency sub-model to obtain the real-time dependency output for each component. Subsequently, these outputs are weighted and summed to obtain the final comprehensive component disk dependency. : ; In this formula, This indicates the overall component disk dependency of the target disk under the current business load; The weighting coefficients represent the disk I / O dependencies of each component.
[0084] In an optional embodiment, the weights corresponding to the output results are determined by the workload proportion of the corresponding application components, or obtained through end-to-end model learning.
[0085] It should be noted that the workload proportion of each application component can be determined by the ratio of its IO utilization to the total IO utilization on the target disk, thus determining the weighting coefficient of its disk IO dependency. Furthermore, the weighting coefficient of disk IO dependency can be considered a learnable parameter, and the disk IO dependency model (including each disk IO dependency sub-model) can be used to... and weighting coefficients This is integrated with the subsequent comprehensive evaluation model for the health status of disk services; the integrated model takes into account the collected samples of all application components regarding the first indicator item, and... The corresponding hardware parameter feature set outputs a comprehensive evaluation value of the business performance indicators.
[0086] The disk I / O dependency model described above can accurately quantify the I / O dependency characteristics of a single component on the disk, and can also adapt the weights to the actual scenarios of multiple components running together, providing characteristic inputs that fit the business load for the comprehensive evaluation model of disk business performance indicators, thus ensuring the accuracy of disk business health status assessment.
[0087] In an optional embodiment, the comprehensive evaluation model for disk service performance indicators is constructed through the following steps: Obtain a sample set of business performance metrics for the target disk and assign weights to it to obtain a set of disk business performance evaluation labels after merging the metrics; Based on the disk service performance evaluation tag set and the corresponding third comprehensive component disk dependency set, obtain the service health status evaluation tag set of the target disk; Based on the business health status evaluation label set and the corresponding third hardware parameter feature set, a multivariate linear regression fitting is performed to obtain a comprehensive evaluation model for the disk business performance indicators.
[0088] Furthermore, when merging indicators, the weight of each indicator to be weighted is determined by the dispersion of the individual indicator data and the total dispersion of all indicator data.
[0089] It should be noted that the comprehensive evaluation model for disk service performance metrics is a mathematical model that combines application component dependencies and directly and accurately infers the health status of the service through disk hardware parameters (SMART metrics). The specific construction process is as follows: The first step is to merge the independent business performance indicators of each sample to obtain the disk business performance evaluation label set: First, based on actual operational experience with resource pools / data centers, we define the most commonly used set of metrics for the impact of disks on business. {Disk I / O utilization, disk I / O wait time}, where disk I / O utilization is the total utilization, used to reflect the overall busyness of the disk.
[0090] Obtain a sample set of business performance metrics for the target disk over a historical period. The set contains n samples, each sample is composed of... constitute.
[0091] Next, the business performance indicator sample set was analyzed. Normalization is performed to map the index values to the [0,1] interval, as shown in the following formula: ; In this formula, This represents the j-th index value of the i-th sample after normalization. The j-th index value of the i-th sample before normalization; and These are the minimum and maximum values of the j-th indicator across all samples, respectively.
[0092] Subsequently, the dispersion of each normalized indicator is calculated, and weights are assigned accordingly. These weights are not subjectively set, but are determined by the dispersion of individual indicator data and the overall dispersion of all indicator data. The specific calculation formula is as follows: ; In this formula, j represents the j-th indicator; m is the total number of indicators; and n is the total number of samples. This represents the j-th index value of the i-th sample after normalization. The sample average value of the j-th indicator; The weight of the j-th indicator reflects the principle that "the more dispersed the distribution of individual indicator data values (the greater the degree of dispersion), the greater its information content, and the higher its weight should be."
[0093] Using the aforementioned weights, a comprehensive evaluation label value for disk service performance is obtained for each sample through linear weighted merging. The calculation formula for any sample is: ; In this formula, and These are the weighting coefficients; This represents the normalized disk I / O utilization rate. This is the normalized disk I / O wait time; These are the disk service performance evaluation labels for the corresponding samples.
[0094] The second step is to construct a set of labels for evaluating the health status of the business: By combining the application components' dependence on the disk, the disk's business performance evaluation tags are associated with the overall component's disk dependence, resulting in a tag set that reflects the actual business health impact of the disk (i.e., the business health status evaluation tag set).
[0095] Application component runtime data collected concurrently with the business performance sample set was used to calculate the disk dependency set of the third comprehensive component using a disk I / O dependency model; its value was... This reflects the overall dependence of all application components on disk I / O resources.
[0096] The disk service performance evaluation labels obtained from the previous text will be used to evaluate the disk services performance of each sample. Disk dependency of the corresponding third integrated component Multiply by each other to obtain the business health status evaluation label. This label reflects the same degree of disk service performance anomaly, and for business scenarios with high IO dependence, the impact on their business health is more severe.
[0097] The third step is to construct a comprehensive evaluation model for disk service performance indicators through multiple linear regression fitting: First, based on the third hardware parameter feature set (corresponding to...) Using the feature space as input and the business health status evaluation label set as output, a multiple linear regression model (i.e., disk business health status evaluation model) is constructed: ; In this formula, This is a sample's disk service performance evaluation label. ; This corresponds to the disk dependency of the third integrated component. ; These are the business health status evaluation labels for the samples; This is the bias value; These are regression coefficients, each corresponding to a feature of a hardware parameter; Corresponding to Hardware parameter features in feature space; for conduct Transformation aims to make the data distribution closer to a normal distribution, thereby improving the model's fitting effect.
[0098] The above disk service health status evaluation model is transformed to obtain a comprehensive evaluation model of disk service performance indicators, namely... ; use express and use express This allows us to obtain a comprehensive evaluation model for disk service performance indicators in another display format. .
[0099] It is worth noting that the loss function of the above comprehensive evaluation model... The definition is as follows: ; In this formula, m is the number of samples; n is the feature dimension; It is the predicted value of the j-th sample; The i-th feature value of the j-th sample (i.e., after...) Transformed hardware parameter characteristics); It is the average of all weight coefficients.
[0100] The corresponding gradient descent formula is as follows: In this formula, The learning rate; It is the result of the kth iteration of the i-th model parameter; The i-th feature value of the j-th sample.
[0101] The model is iteratively trained based on the gradient descent formula until the loss function converges, thereby determining the optimal model parameters and ultimately constructing a comprehensive evaluation model for disk service performance indicators based on multiple regression. The specific training process is as follows: the maximum number of iterations is set to 5000, the loss function convergence threshold is 0.01, and the learning rate is... =0.001. During training, if the model's loss value falls below a set threshold after multiple iterations, the model is considered converged, training can be terminated early, and the current parameters are output as the optimal solution. If the loss value remains above the threshold, the remaining iterations continue until the maximum number of iterations is reached. If the model has not converged after reaching the maximum number of iterations, hyperparameters such as the learning rate and regularization parameters can be further adjusted, and a new round of training can be started. This process is repeated until the optimal model parameters that meet the preset performance requirements are obtained. Finally, the trained parameters are substituted into the model to construct a comprehensive evaluation model for the health status of disk services that can be used for inference.
[0102] After completing the construction of the comprehensive evaluation model for disk service health status, the following section will elaborate on the disk failure prediction process (i.e., hardware anomaly detection and service health detection).
[0103] In an optional embodiment, the step of performing hardware anomaly detection on the second hardware parameter feature set within the second time period using the histogram probability density model includes: Based on the histogram probability density model, the first anomaly detection score set corresponding to the first hardware parameter feature set and the second anomaly detection score set corresponding to the second hardware parameter feature set are obtained respectively. Based on the first anomaly detection score set, the first outlier mean and the corresponding outlier difference are calculated to determine the second anomaly threshold. The second outlier mean was calculated based on the second anomaly detection score set. The hardware anomaly detection result is obtained by comparing the second outlier mean and the second anomaly threshold.
[0104] It should be noted that for each sample in the first hardware parameter feature set (i.e., the training set), the degree of deviation from the normal distribution is calculated using a histogram probability density model to obtain the first anomaly detection score set. This set reflects the HIOD score distribution of disks under historical normal conditions. The second set of hardware parameter features to be detected (i.e., the real-time sample set) is input into the same histogram probability density model to calculate the anomaly detection score for each real-time sample, thus obtaining the second anomaly detection score set. .
[0105] In this embodiment of the invention, the determination of the degree of anomaly does not rely on a fixed threshold, but is dynamically calculated based on the disk's own historical behavior, that is, using the first anomaly detection score set. Based on this, the outlier statistic is calculated to determine the second outlier threshold used to judge the degree of anomaly in real-time data, as follows: An outlier mean calculation formula specifically optimized for discrete points is used to reduce the impact of extreme values on the statistical results. The formula is as follows: ; In this formula, It is the first outlier mean; It is the i-th anomaly score in the first anomaly detection score set; It is the total number of samples in the first anomaly detection score set; through logarithmic transformation This makes the data distribution closer to a normal distribution and smooths outliers.
[0106] Based on the first outlier mean The outlier variance (dispersion of outlier scores) is calculated to measure the dispersion of outlier scores in historical normal data. The formula is as follows: .
[0107] Combined with the first outlier mean With the corresponding outlier Determine the second abnormal threshold Among them, the coefficient It can be adjusted according to the sensitivity of actual business needs (e.g., Corresponding to the "sub-health" state, (Corresponding to the "risk" status).
[0108] Calculate the second anomaly detection score set Second outlier mean Then the second outlier mean With the second abnormal threshold If a comparison is made, > If the hardware anomaly detection result is positive, the result is determined to be "suspected anomaly," triggering subsequent serial verification of business health status; otherwise, the hardware anomaly detection result is determined to be "normal / not suspected anomaly."
[0109] In an optional embodiment, comparing the first anomaly threshold and the detection mean to obtain the service health status of the suspected abnormal disk includes: When the average detection value is not greater than the first abnormal threshold, the business health status is a non-abnormal status; When the average detection value is greater than the first abnormal threshold, the business health status is abnormal, and the suspected abnormal disk is determined to have an anomaly.
[0110] It should be noted that the embodiments of the present invention improve the accuracy of disk failure prediction by using a closed-loop verification of hardware anomaly detection and business performance anomaly detection (i.e., business health detection) to avoid misjudgment from a single dimension.
[0111] Based on the disk dependency set of the first comprehensive component and the first hardware parameter feature set of the first time period, the first comprehensive evaluation set of business performance indicators is calculated through the comprehensive evaluation model of disk business performance indicators. Obtain the first comprehensive evaluation set. root mean square and standard deviation .
[0112] Based on the second comprehensive component disk dependency set and the second hardware parameter feature set for the second time period, a second comprehensive evaluation set of business performance indicators is calculated using the same comprehensive evaluation model. Obtain the second comprehensive evaluation set. The root mean square of the sample is used to obtain the detection mean. .
[0113] According to the anomaly detection formula The health status of disk services can be determined; among them, This is the verification value for anomaly detection; This is a multiple of the standard deviation, usually taken as 3. If >1 indicates Overall distance The average value is far from reaching If the deviation is more than one standard deviation, it can be judged as abnormal business performance.
[0114] Based on this, a first anomaly threshold can be defined according to the anomaly detection formula. equal + In other words, if >1 (i.e., the test mean) > If the business health status is abnormal, then the suspected abnormal disk is confirmed to be faulty and requires immediate disk maintenance or replacement; if ≤1 (i.e., the test mean) ≤ If the condition is not met, the business health status is normal, and there may be temporary fluctuations in hardware parameters, which do not require intervention.
[0115] In summary, the disk anomaly detection method provided in this invention, combined with the actual business needs of online anomaly detection of massive disks in resource pools / data centers, proposes to define disk health status from the perspective of its impact on business system operation, starting from practical operation and maintenance. It comprehensively considers the actual impact of disk business performance indicators (such as disk IO utilization and IO wait time) on upper-layer businesses, as well as the resource dependence of different application components on disk IO, and deeply integrates this with the disk hardware parameter SMART index to build a correlation analysis model and anomaly detection model (hardware anomaly detection and business health detection) based on machine learning technology for each disk. Compared to simply judging the physical state of the disk, the business health status of this invention can more sensitively and directly reflect the actual impact of the disk on the business system, and therefore can serve as the "gold standard" for disk fault assessment in the complex and ever-changing business scenarios of resource pools / data centers.
[0116] This invention ultimately achieves accurate online anomaly detection of disks in resource pool / data center scenarios, with significant advantages: First, it is highly targeted, enabling individualized and accurate monitoring of disks; second, it has high detection accuracy, integrating multi-dimensional information for comprehensive judgment; third, it is timely, eliminating the need to expend significant effort collecting negative sample data from disks; the overall solution is lightweight and easy to implement, and can be quickly adapted to data center operation and maintenance scenarios of various sizes, effectively supporting the upgrade from "passive emergency repair" to "proactive prediction" operation and maintenance mode.
[0117] See Figure 3 This is a schematic diagram of an embodiment of the disk anomaly detection device provided by the present invention.
[0118] A second aspect of the present invention provides a disk anomaly detection device, comprising: The hardware anomaly detection module 11 is used to construct a histogram probability density model based on the first hardware parameter feature set of the target disk in the first time period, and to perform hardware anomaly detection on the second hardware parameter feature set in the second time period through the histogram probability density model. The abnormal threshold acquisition module 12 is used to input the first comprehensive component disk dependency set and the first hardware parameter feature set of the suspected abnormal disk in the first time period into the comprehensive evaluation model of disk business performance indicators when the hardware abnormality detection result indicates that the target disk is a suspected abnormal disk, so as to obtain the first comprehensive evaluation set of business performance indicators and determine the first abnormal threshold. The detection mean acquisition module 13 is used to input the second comprehensive component disk dependency set and the second hardware parameter feature set of the suspected abnormal disk in the second time period into the comprehensive evaluation model to obtain the second comprehensive evaluation set of business performance indicators and calculate the detection mean. The health status acquisition module 14 is used to obtain the business health status of suspected abnormal disks by comparing the first abnormal threshold with the detection mean.
[0119] It should be noted that the disk anomaly detection device provided in the second aspect embodiment of the present invention can implement all the processes of the disk anomaly detection method described in any of the first aspect embodiments. The functions and technical effects of each module and unit in the device are the same as those of the disk anomaly detection method described in any of the first aspect embodiments, and will not be repeated here.
[0120] A third aspect of the present invention provides a computer-readable storage medium comprising a stored computer program; wherein, when the computer program is executed, it controls the device in which the computer-readable storage medium is located to perform the disk anomaly detection method described in any embodiment of the first aspect.
[0121] A fourth aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the disk anomaly detection method described in any embodiment of the first aspect.
[0122] See Figure 4 This is a schematic diagram of an embodiment of the electronic device provided by the present invention.
[0123] A fifth aspect of the present invention provides an electronic device including a processor 21, a memory 22, and a computer program stored in the memory 22 and configured to be executed by the processor 21, wherein the processor, when executing the computer program, implements the disk anomaly detection method described in any embodiment of the first aspect.
[0124] Preferably, the computer program can be divided into one or more modules / units (such as computer program one, computer program two, ...), and the one or more modules / units are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.
[0125] The processor 21 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 21 can be any conventional processor. The processor 21 is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines.
[0126] The memory 22 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 22 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, and a flash card, etc., or the memory 22 can also be other volatile solid-state storage devices.
[0127] It should be noted that the aforementioned electronic devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 4 The structural block diagram shown is merely a structural example of the above-described electronic device and does not constitute a limitation on the structure of the above-described electronic device. The above-described electronic device may include more or fewer components than shown, or combine certain components, or different components.
[0128] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A disk anomaly detection method, characterized in that, include: Based on the first hardware parameter feature set of the target disk in the first time period, a histogram probability density model is constructed, and hardware anomaly detection is performed on the second hardware parameter feature set in the second time period through the histogram probability density model. When the hardware anomaly detection result indicates that the target disk is a suspected abnormal disk, the first comprehensive component disk dependency set and the first hardware parameter feature set of the suspected abnormal disk in the first time period are input into the comprehensive evaluation model of disk business performance indicators to obtain the first comprehensive evaluation set of business performance indicators, so as to determine the first anomaly threshold. The second comprehensive component disk dependency set and the second hardware parameter feature set of the suspected abnormal disk in the second time period are input into the comprehensive evaluation model to obtain the second comprehensive evaluation set of business performance indicators and calculate the detection mean. The business health status of suspected abnormal disks is obtained by comparing the first abnormal threshold with the detection mean.
2. The disk anomaly detection method as described in claim 1, characterized in that, The method further includes: When the hardware anomaly detection result indicates that the target disk is not a suspected anomaly disk, the detection of business health status is stopped, and the hardware anomaly detection result is output directly.
3. The disk anomaly detection method as described in claim 1, characterized in that, The first hardware parameter feature set and the second hardware parameter feature set both contain feature items composed of representative comprehensive indicators and weakly correlated hardware parameter indicators; wherein, the comprehensive indicators are obtained by principal component analysis to reduce the dimensionality of highly correlated hardware parameter indicators.
4. The disk anomaly detection method as described in claim 1, characterized in that, The step of detecting hardware anomalies by using the histogram probability density model to analyze the second hardware parameter feature set within the second time period includes: Based on the histogram probability density model, the first anomaly detection score set corresponding to the first hardware parameter feature set and the second anomaly detection score set corresponding to the second hardware parameter feature set are obtained respectively. Based on the first anomaly detection score set, the first outlier mean and the corresponding outlier difference are calculated to determine the second anomaly threshold. The second outlier mean was calculated based on the second anomaly detection score set. The hardware anomaly detection result is obtained by comparing the second outlier mean and the second anomaly threshold.
5. The disk anomaly detection method as described in claim 1, characterized in that, Both the first integrated component disk dependency set and the second integrated component disk dependency set are calculated using a disk I / O dependency model; wherein, the disk I / O dependency model is constructed through the following steps: Obtain the historical sample set of the first indicator item when each application component runs independently, and train the disk I / O dependency sub-model of the corresponding application component based on each historical sample set; When application components run together, the collected data of each application component regarding the first indicator item are input into the corresponding disk I / O dependency sub-model. The output results of each disk I / O dependency sub-model are weighted to obtain the disk I / O dependency model.
6. The disk anomaly detection method as described in claim 5, characterized in that, The weights corresponding to the output results are determined by the workload proportion of the corresponding application components, or obtained through end-to-end model learning.
7. The disk anomaly detection method as described in claim 5, characterized in that, The first indicator includes: the corresponding component performance indicators, server performance indicators, and disk performance indicators.
8. The disk anomaly detection method as described in claim 1, characterized in that, The comprehensive evaluation model for disk service performance indicators is constructed through the following steps: Obtain a sample set of business performance metrics for the target disk and assign weights to it to obtain a set of disk business performance evaluation labels after merging the metrics; Based on the disk service performance evaluation tag set and the corresponding third comprehensive component disk dependency set, obtain the service health status evaluation tag set of the target disk; Based on the business health status evaluation label set and the corresponding third hardware parameter feature set, a multivariate linear regression fitting is performed to obtain a comprehensive evaluation model for the disk business performance indicators.
9. The disk anomaly detection method as described in claim 8, characterized in that, When merging indicators, the weight of each indicator to be weighted is determined by the dispersion of the individual indicator data and the total dispersion of all indicator data.
10. The disk anomaly detection method as described in claim 1, characterized in that, The first time period is updated on a rolling basis according to a preset time cycle; wherein, the first time period is earlier than the second time period.
11. The disk anomaly detection method as described in claim 1, characterized in that, The step of comparing the first anomaly threshold and the detection mean to obtain the service health status of the suspected abnormal disk includes: When the average detection value is not greater than the first abnormal threshold, the business health status is a non-abnormal status; When the average detection value is greater than the first abnormal threshold, the business health status is abnormal, and the suspected abnormal disk is determined to have an anomaly.
12. A disk anomaly detection device, characterized in that, include: The hardware anomaly detection module is used to construct a histogram probability density model based on the first hardware parameter feature set of the target disk in the first time period, and to perform hardware anomaly detection on the second hardware parameter feature set in the second time period through the histogram probability density model. The anomaly threshold acquisition module is used to input the first comprehensive component disk dependency set and the first hardware parameter feature set of the suspected abnormal disk within the first time period into the comprehensive evaluation model of disk business performance indicators when the hardware anomaly detection result indicates that the target disk is a suspected abnormal disk, so as to obtain the first comprehensive evaluation set of business performance indicators and determine the first anomaly threshold. The detection mean acquisition module is used to input the second comprehensive component disk dependency set and the second hardware parameter feature set of the suspected abnormal disks in the second time period into the comprehensive evaluation model to obtain the second comprehensive evaluation set of business performance indicators and calculate the detection mean. The health status acquisition module is used to obtain the business health status of suspected abnormal disks by comparing the first abnormal threshold with the detection average.
13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the disk anomaly detection method as described in any one of claims 1 to 11.
14. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the disk anomaly detection method as described in any one of claims 1 to 11.
15. An electronic device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the disk anomaly detection method as described in any one of claims 1 to 11.