A disk failure prediction method and device for processing extremely unbalanced data
Through the combination of density clustering algorithm and neural network, extremely unbalanced data are processed, and the noise amplification and overfitting problems caused by undersampling and oversampling in existing disk failure prediction methods are solved, achieving more accurate disk failure prediction .
Patent Information
- Application Number
- CN202211720060.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-12-30
AI Technical Summary
When existing disk failure prediction methods deal with extremely unbalanced data, undersampling and oversampling methods have problems such as amplifying noise, increasing the risk of overfitting, blurring the boundaries of positive and negative samples, and increasing the difficulty of classification.
A feature processing method based on density clustering algorithm is adopted to preprocess, feature extraction, cluster and undersample disk log data, a more balanced feature set is generated, and a neural network is used to predict faults.
It improves the accuracy of disk failure prediction and model stability, reduces the risk of overfitting, enhances the processing ability of extremely unbalanced data, and can predict disk failures more accurately in a real production environment.
Smart Images

Figure CN115938406B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of software development technology, and in particular relates to a disk failure prediction method and device for processing extremely unbalanced data. Background Art
[0002] With the advent of the big data era, the value of digital data is increasing. Most of the world's data is stored on disks. This massive data storage method poses a huge challenge to the stability of disks. Once a disk fails, the stored data may be lost forever. Based on this, how to accurately predict disk failures is of great significance to improving the reliability of data centers.
[0003] Based on an annual failure rate of 1%, the ratio of failed disks to normal disks in a one-year data set is about 1:100, and the records of failed disks are abnormal only on the day of the failure, while the records before the failure are all normal records, which greatly aggravates the imbalance of the data. In the Backblaze public data set, the ratio of failed records to normal records is about 1:30,000, which is an extreme imbalance problem.
[0004] At present, disk failure prediction methods usually use undersampling and oversampling methods to deal with this kind of extremely unbalanced data problem. However, simple oversampling simply repeats the positive examples, which has the risk of amplifying noise and increasing the degree of overfitting. For oversampling methods such as SMOTE, it is easy to produce distribution marginalization problems, which blurs the boundaries between positive and negative samples and increases the difficulty of classification. For simple undersampling algorithms, most of the negative examples (multiple examples) are discarded, resulting in large model deviation. For some improved undersampling algorithms, multiple examples are divided into N non-overlapping parts, which are combined with fewer examples, trained N models and combined. Although this algorithm does not cause information loss, it is expensive to train multiple models and there is also a risk of overfitting. Summary of the invention
[0005] In order to solve the technical defects of the above-mentioned disk failure prediction methods at the current stage, such as the risk of amplifying noise by undersampling and oversampling methods, increasing the degree of overfitting, blurring the boundary between positive and negative samples, increasing the difficulty of classification, and the risk of overfitting, this application proposes a disk failure prediction method and device for processing extremely unbalanced data, and the technical solution is as follows:
[0006] In a first aspect, an embodiment of the present application provides a disk failure prediction method for processing extremely unbalanced data, comprising:
[0007] Obtain the log data of the disk and preprocess the log data of the disk;
[0008] Performing segmentation processing on the log data of the disk after preprocessing based on a preset time interval, and extracting a first feature set from the log data of the disk after segmentation processing;
[0009] Performing clustering processing on the first feature set based on a density clustering algorithm, and performing undersampling processing on the first feature set after the clustering processing to obtain a second feature set;
[0010] A feature set of the disk at the next moment is predicted according to the second feature set, and fault information of the disk is determined based on the feature set of the disk at the next moment.
[0011] In an optional solution of the first aspect, preprocessing the log data of the disk includes:
[0012] Determine whether there is any missing data in the log data of the disk;
[0013] When it is determined that there are missing data in the log data of the disk, the missing data is interpolated by the mean value;
[0014] Normalize the log data of the disk after mean interpolation.
[0015] In yet another optional solution of the first aspect, extracting a first feature set from log data of the disk after the partitioning process includes:
[0016] Determine the feature values belonging to the same attribute from the log data of the disk corresponding to each time interval, and calculate the weight feature value of each attribute according to the preset weight and each feature value belonging to the same attribute;
[0017] Determine the feature values belonging to the same attribute from the log data of the disk corresponding to any two consecutive time intervals, and perform difference calculation on the two feature values belonging to the same attribute to calculate the difference feature value of each attribute;
[0018] The weight feature value of each attribute and the difference feature value of each attribute are taken as the first feature set.
[0019] In yet another optional solution of the first aspect, before taking the weight feature value of each attribute and the difference feature value of each attribute as the first feature set, the method further includes:
[0020] Determine the maximum eigenvalue of each attribute and the minimum eigenvalue of each attribute from each eigenvalue belonging to the same attribute;
[0021] Calculate the mean of each eigenvalue belonging to the same attribute to obtain the mean eigenvalue of each attribute;
[0022] Calculate the variance of each eigenvalue belonging to the same attribute to obtain the variance eigenvalue of each attribute;
[0023] The weight feature value of each attribute and the difference feature value of each attribute are used as the first feature set, including:
[0024] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute are taken as the first feature set.
[0025] In yet another optional solution of the first aspect, before taking the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute as the first feature set, the method further includes:
[0026] Determine the characteristic values belonging to the same attribute from the log data of the disk corresponding to any two consecutive time intervals, and calculate the slope according to the coordinates corresponding to the two characteristic values belonging to the same attribute to obtain the slope characteristic value of each attribute;
[0027] Determine the number of eigenvalues whose values are in continuous growth among each eigenvalue belonging to the same attribute, and use the number of eigenvalues as the continuously growing eigenvalue of each attribute;
[0028] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute are used as the first feature set, including:
[0029] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute and the continuous growth eigenvalue of each attribute are taken as the first feature set.
[0030] In yet another optional solution of the first aspect, before taking the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, and the continuous growth eigenvalue of each attribute as the first feature set, the method further includes:
[0031] Determining a characteristic value of the first attribute and a characteristic value of the second attribute in log data of the disk corresponding to each time interval;
[0032] Combining and calculating the characteristic value of the first attribute and the characteristic value of the second attribute according to a preset combination method to obtain a combined characteristic value;
[0033] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, and the continuous growth eigenvalue of each attribute are taken as the first feature set, including:
[0034] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, the continuous growth eigenvalue of each attribute and the combined eigenvalue are taken as the first feature set.
[0035] In another optional solution of the first aspect, clustering is performed on the first feature set based on a density clustering algorithm, and undersampling is performed on the first feature set after clustering to obtain a second feature set, including:
[0036] Performing clustering processing on the first feature set based on a density clustering algorithm to obtain at least two clusters; wherein each cluster includes at least two feature values;
[0037] Eliminate all feature values included in at least two clusters in the first feature set to obtain a third feature set;
[0038] At least two clusters and the third feature set are merged according to a preset ratio to obtain a second feature set.
[0039] In yet another optional solution of the first aspect, predicting a feature set of the disk at the next moment according to the second feature set includes:
[0040] Inputting the second feature set into the trained first neural network to predict the feature set corresponding to the faulty disk;
[0041] Determine a sample feature set having the same number as the feature set corresponding to the faulty disk, and divide and process the feature set corresponding to the faulty disk and the sample feature set according to a preset time interval;
[0042] The feature set and sample feature set corresponding to the faulty disk after the division processing are encoded, and the feature set and sample feature set corresponding to the faulty disk after the encoding processing are input into the trained second neural network to predict the feature set of the disk at the next moment.
[0043] In a second aspect, an embodiment of the present application provides a disk failure prediction device for processing extremely unbalanced data, comprising:
[0044] The log processing module is used to obtain the log data of the disk and pre-process the log data of the disk;
[0045] A feature extraction module, used for performing segmentation processing on the log data of the disk after preprocessing based on a preset time interval, and extracting a first feature set from the log data of the disk after segmentation processing;
[0046] A feature processing module, used for performing clustering processing on the first feature set based on a density clustering algorithm, and performing undersampling processing on the first feature set after the clustering processing to obtain a second feature set;
[0047] The model prediction module is used to predict the feature set of the disk at the next moment according to the second feature set, and determine the fault information of the disk based on the feature set of the disk at the next moment.
[0048] In an optional solution of the second aspect, the log processing module includes:
[0049] Determine whether there is any missing data in the log data of the disk;
[0050] When it is determined that there are missing data in the log data of the disk, the missing data is interpolated by the mean value;
[0051] Normalize the log data of the disk after mean interpolation.
[0052] In yet another optional solution of the second aspect, the feature extraction module includes:
[0053] Determine the feature values belonging to the same attribute from the log data of the disk corresponding to each time interval, and calculate the weight feature value of each attribute according to the preset weight and each feature value belonging to the same attribute;
[0054] Determine the feature values belonging to the same attribute from the log data of the disk corresponding to any two consecutive time intervals, and perform difference calculation on the two feature values belonging to the same attribute to calculate the difference feature value of each attribute;
[0055] The weight feature value of each attribute and the difference feature value of each attribute are taken as the first feature set.
[0056] In yet another optional solution of the second aspect, the feature extraction module further includes:
[0057] Determine the maximum eigenvalue of each attribute and the minimum eigenvalue of each attribute from each eigenvalue belonging to the same attribute;
[0058] Calculate the mean of each eigenvalue belonging to the same attribute to obtain the mean eigenvalue of each attribute;
[0059] Calculate the variance of each eigenvalue belonging to the same attribute to obtain the variance eigenvalue of each attribute;
[0060] The weight feature value of each attribute and the difference feature value of each attribute are used as the first feature set, including:
[0061] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute are taken as the first feature set.
[0062] In yet another optional solution of the second aspect, the feature extraction module further includes:
[0063] Determine the characteristic values belonging to the same attribute from the log data of the disk corresponding to any two consecutive time intervals, and calculate the slope according to the coordinates corresponding to the two characteristic values belonging to the same attribute to obtain the slope characteristic value of each attribute;
[0064] Determine the number of eigenvalues whose values are continuously increasing in each eigenvalue belonging to the same attribute, and use the number of eigenvalues as the continuously increasing eigenvalue of each attribute;
[0065] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute are used as the first feature set, including:
[0066] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute and the continuous growth eigenvalue of each attribute are taken as the first feature set.
[0067] In yet another optional solution of the second aspect, the feature extraction module further includes:
[0068] Determining a characteristic value of the first attribute and a characteristic value of the second attribute in log data of the disk corresponding to each time interval;
[0069] Combining and calculating the characteristic value of the first attribute and the characteristic value of the second attribute according to a preset combination method to obtain a combined characteristic value;
[0070] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, and the continuous growth eigenvalue of each attribute are taken as the first feature set, including:
[0071] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, the continuous growth eigenvalue of each attribute and the combined eigenvalue are taken as the first feature set.
[0072] In yet another optional solution of the second aspect, the feature processing module includes:
[0073] Performing clustering processing on the first feature set based on a density clustering algorithm to obtain at least two clusters; wherein each cluster includes at least two feature values;
[0074] Eliminate all feature values included in at least two clusters in the first feature set to obtain a third feature set;
[0075] At least two clusters and the third feature set are merged according to a preset ratio to obtain a second feature set.
[0076] In yet another optional solution of the second aspect, the model prediction module includes:
[0077] The second feature set is input into the trained first neural network to predict the feature set corresponding to the faulty disk;
[0078] Determine a sample feature set having the same number as the feature set corresponding to the faulty disk, and divide and process the feature set corresponding to the faulty disk and the sample feature set according to a preset time interval;
[0079] The feature set and sample feature set corresponding to the faulty disk after the division processing are encoded, and the feature set and sample feature set corresponding to the faulty disk after the encoding processing are input into the trained second neural network to predict the feature set of the disk at the next moment.
[0080] In a third aspect, an embodiment of the present application further provides a disk failure prediction device for processing extremely unbalanced data, including a processor and a memory;
[0081] The processor is connected to the memory;
[0082] A memory for storing executable program codes;
[0083] The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the disk failure prediction method for processing extremely unbalanced data provided by the first aspect of the embodiment of the present application or any implementation method of the first aspect.
[0084] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the disk failure prediction method for processing extremely unbalanced data provided by the first aspect of the embodiment of the present application or any one of the implementation methods of the first aspect can be implemented.
[0085] In an embodiment of the present application, when a disk fault is detected, the log data of the disk can be obtained and the log data of the disk can be preprocessed; the log data of the preprocessed disk can be divided based on a preset time interval, and a first feature set can be extracted from the log data of the divided disk; the first feature set can be clustered based on a density clustering algorithm, and the first feature set after clustering can be undersampled to obtain a second feature set; the feature set of the disk at the next moment can be predicted based on the second feature set, and the fault information of the disk can be determined based on the feature set of the disk at the next moment. By learning the law of disk failure under extremely unbalanced data and predicting disk failures more accurately in a real production environment, the planning of operation and maintenance work can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0086] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0087] Figure 1 An overall flow chart of a disk failure prediction method for processing extremely unbalanced data provided by an embodiment of the present application;
[0088] Figure 2 A schematic diagram of the effect of obtaining a second feature set provided in an embodiment of the present application;
[0089] Figure 3 A schematic diagram of the training effect of a first neural network provided in an embodiment of the present application;
[0090] Figure 4 A schematic diagram of the training effect of a second neural network provided in an embodiment of the present application;
[0091] Figure 5A schematic diagram of the structure of a disk failure prediction device for processing extremely unbalanced data provided by an embodiment of the present application;
[0092] Figure 6 A schematic diagram of the structure of another disk failure prediction device for processing extremely unbalanced data provided in an embodiment of the present application. DETAILED DESCRIPTION
[0093] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0094] In the following introduction, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The following introduction provides multiple embodiments of the present application, and different embodiments can be replaced or combined, so the present application can also be considered to include all possible combinations of the same and / or different embodiments recorded. Therefore, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then the present application should also be considered to include embodiments containing one or more of all other possible combinations of A, B, C, and D, although the embodiment may not be clearly recorded in the following text.
[0095] The following description provides examples and does not limit the scope, applicability or examples set forth in the claims. Changes may be made to the functions and arrangements of the elements described without departing from the scope of the present application. Various processes or components may be appropriately omitted, substituted or added to each example. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted or combined. In addition, features described in some examples may be combined in other examples.
[0096] See also Figure 1 , Figure 1 An overall flow chart of a disk failure prediction method for processing extremely unbalanced data provided by an embodiment of the present application is shown.
[0097] like Figure 1 As shown, the disk failure prediction method for processing extremely unbalanced data may include at least the following steps:
[0098] Step 102: Obtain the log data of the disk and pre-process the log data of the disk.
[0099] Specifically, when performing fault detection on the disk, the log data of the disk within a certain period of time can be obtained. The method of obtaining the log data of the disk can be, but is not limited to, reading the SMART attribute information of the disk based on specific software, and determining the log data based on the SMART attribute information. The specific software may include DiskGenius, CrystalDiskInfo, Hard Disk Sentine and other software, and its use process is not described in detail here. It can be understood that this technology of obtaining the SMART attribute information of the disk is a self-monitoring, analysis, and reporting technology commonly used by disks. This technology can detect and record the temperature, circuit, disk, head and other states of the disk when the disk is working, and each state can correspond to different attributes of the disk.
[0100] It should be noted that the log data on the disk has a certain time sequence. For example, when the certain time mentioned above is a week, the log data on the disk may include the log data corresponding to each day of the week, or may include the log data corresponding to each hour of the week. The time intervals are not limited to this.
[0101] Furthermore, after the log data of the disk is obtained, the log data of the disk may be preprocessed to increase the availability and effectiveness of the log data of the disk.
[0102] As an option of the embodiment of the present application, preprocessing the log data of the disk includes:
[0103] Determine whether there is any missing data in the log data of the disk;
[0104] When it is determined that there are missing data in the log data of the disk, mean interpolation processing is performed on the missing data;
[0105] Normalize the log data of the disk after mean interpolation.
[0106] Specifically, in the process of preprocessing the log data of the disk, it is possible to first determine whether there is missing data in the log data of the disk. For example, when it is detected in the log data of the disk that data of a certain attribute does not exist on a certain day or at a certain time, it is determined that there is missing data in the log data of the disk, and the missing data is the data of a certain attribute that does not exist on that certain day or at that time.
[0107] Furthermore, when it is determined that there is missing data in the log data on the disk, the missing data can be interpolated by the mean, wherein the mean interpolation method can be understood as obtaining the data of the attribute corresponding to the missing data in other time periods, and performing mean calculation on all the data of the attribute in other time periods, and using the calculation result as the missing data. For example, if the missing data is the data of a certain attribute on the third day of a week, the mean interpolation method can be to obtain the data of the attribute on the first, second, fourth, and seventh days of a week, and perform mean calculation on the data of the six days, and use the calculation result as the data of the third day to supplement the missing parts in the log data on the disk.
[0108] Furthermore, after the missing data in the log data of the disk is processed by mean interpolation, since the value ranges of different attribute data are quite different, the log data of the disk processed by mean interpolation can be normalized or standardized, but not limited to, to perform feature scaling on the log data of the disk processed by mean interpolation, thereby increasing the availability and effectiveness of the log data. It is understandable that the normalization or standardization method is a conventional technical means for data processing in this field, and will not be elaborated here.
[0109] Step 104 : partition the log data of the disk after the preprocessing based on a preset time interval, and extract a first feature set from the log data of the disk after the partitioning.
[0110] Specifically, after preprocessing the log data of the disk, the preprocessed log data of the disk can be divided and processed based on a preset time interval, wherein the preset time interval can be but is not limited to a 24-hour time interval, that is, the preprocessed disk log data is divided and processed according to days to obtain the preprocessed disk log data corresponding to each day.
[0111] Furthermore, after the log data of the disk that has been preprocessed is divided and processed, the characteristic value belonging to the same attribute can be determined from the log data of the disk corresponding to each time interval. The characteristic value of the attribute can also be understood as the specific value of the attribute in the log data of the disk corresponding to each time interval, and the weight characteristic value of each attribute is calculated according to the preset weight and each characteristic value belonging to the same attribute. For example, taking the characteristic values of attribute A in the log data of the disk corresponding to each time interval as a1, a2, a3 and a4, the preset weights can correspond to 0.1, 0.2, 0.3 and 0.4, respectively, then the weight characteristic value of attribute A can be, but is not limited to, expressed by the following formula:
[0112] a1*0.1+ a2*0.2+ a3*0.3+ a4*0.4
[0113] It is understandable that in the embodiment of the present application, the preset weight corresponding to each attribute remains consistent, and this is not limited to this.
[0114] Furthermore, the characteristic values belonging to the same attribute can be determined from the log data of the disk corresponding to any two consecutive time intervals, and the difference calculation of the two characteristic values belonging to the same attribute can be performed to obtain the difference characteristic value of each attribute. For example, the characteristic values of attribute A in the log data of the disk corresponding to each time interval can be expressed as a1, a2, a3 and a4, respectively, where a1 and a2 are two characteristic values determined in the log data of the disk corresponding to two consecutive time intervals, a2 and a3 are two characteristic values determined in the log data of the disk corresponding to two consecutive time intervals, and a3 and a4 are two characteristic values determined in the log data of the disk corresponding to two consecutive time intervals. Then, the difference between a1 and a2, the difference between a2 and a3, and the difference between a3 and a4 can be calculated respectively, and all the differences are used as the difference characteristic values of attribute A.
[0115] Furthermore, the set of weighted eigenvalues and difference eigenvalues of each attribute calculated above can be used as the first feature set, wherein the weighted eigenvalues and difference eigenvalues of each attribute can be understood as window features in the log data of the disk, so as to avoid problems such as large noise and little valuable information in the log data.
[0116] As another option of the embodiment of the present application, before taking the weight feature value of each attribute and the difference feature value of each attribute as the first feature set, the method further includes:
[0117] Determine the maximum eigenvalue of each attribute and the minimum eigenvalue of each attribute from each eigenvalue belonging to the same attribute;
[0118] Calculate the mean of each eigenvalue belonging to the same attribute to obtain the mean eigenvalue of each attribute;
[0119] Calculate the variance of each eigenvalue belonging to the same attribute to obtain the variance eigenvalue of each attribute;
[0120] The weight feature value of each attribute and the difference feature value of each attribute are used as the first feature set, including:
[0121] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute are taken as the first feature set.
[0122] Specifically, before taking the weighted eigenvalue of each attribute and the difference eigenvalue of each attribute as the first feature set, the maximum eigenvalue and the minimum eigenvalue of each attribute can also be determined from the eigenvalues belonging to the same attribute. For example, the eigenvalues of attribute A in the log data of the disk corresponding to each time interval can be expressed as a1, a2, a3 and a4 respectively, where a1 is less than a2, a2 is less than a3, and a3 is less than a4. Then the maximum eigenvalue of attribute A is a4, and the minimum eigenvalue of attribute A is a1.
[0123] Furthermore, the mean of each eigenvalue belonging to the same attribute can be calculated to obtain the mean eigenvalue of each attribute, and the variance of each eigenvalue belonging to the same attribute can be calculated to obtain the variance eigenvalue of each attribute, wherein the mean calculation method and the variance calculation method are conventional statistical techniques and will not be elaborated here.
[0124] After obtaining the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute, the weight eigenvalue of each attribute and the difference eigenvalue of each attribute mentioned above can be combined to take the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute as the first feature set. The maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute can be understood as statistical features in the log data of the disk, so as to further avoid the problems of large noise and little valuable information in the log data.
[0125] As another option of the embodiment of the present application, before taking the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute as the first feature set, it also includes:
[0126] Determine the characteristic values belonging to the same attribute from the log data of the disk corresponding to any two consecutive time intervals, and calculate the slope according to the coordinates corresponding to the two characteristic values belonging to the same attribute to obtain the slope characteristic value of each attribute;
[0127] Determine the number of eigenvalues whose values are in continuous growth among each eigenvalue belonging to the same attribute, and use the number of eigenvalues as the continuously growing eigenvalue of each attribute;
[0128] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute are used as the first feature set, including:
[0129] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute and the continuous growth eigenvalue of each attribute are taken as the first feature set.
[0130] Specifically, before taking the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute and the variance eigenvalue of each attribute as the first feature set, the number of eigenvalues in continuous growth can also be determined in each eigenvalue belonging to the same attribute, and the number of eigenvalues in continuous growth can be used as the continuous growth eigenvalue of each attribute. For example, the eigenvalues of attribute A in the log data of the disk corresponding to each time interval can be represented as a1, a2, a3 and a4, respectively, where a1 is less than a2, a2 is less than a3, and a3 is greater than a4, then a1, a2 and a3 can be determined as three continuously growing eigenvalues, and then the continuously growing eigenvalue of attribute A is 3. It can be understood that in the embodiment of the present application, the number of eigenvalues in continuous decline and the number of eigenvalues in a flat state can also be determined in combination with each eigenvalue belonging to the same attribute, which is not limited to this here.
[0131] Furthermore, the characteristic values belonging to the same attribute can be determined in the log data corresponding to any two consecutive time intervals, and the two characteristic values can be expressed in the form of coordinates, wherein the horizontal coordinate of the coordinates can be but not limited to the time intervals corresponding to the two characteristic values, and the vertical coordinate of the coordinates can be but not limited to the numerical values corresponding to the two characteristic values. Then, the slope of the two characteristic values can be calculated by the slope calculation formula, and the slope can be used as the slope characteristic value of each attribute.
[0132] Furthermore, after obtaining the continuous growth eigenvalue and slope eigenvalue of each attribute, the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute mentioned above can be combined to take the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, and the continuous growth eigenvalue of each attribute as the first feature set. Among them, the continuous growth eigenvalue and slope eigenvalue of each attribute can be understood as the trend characteristics in the log data of the disk, so as to further avoid the problems of large noise and little valuable information in the log data.
[0133] As another option of the embodiment of the present application, before taking the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, and the continuous growth eigenvalue of each attribute as the first feature set, it also includes:
[0134] Determining a characteristic value of the first attribute and a characteristic value of the second attribute in log data of the disk corresponding to each time interval;
[0135] Combining and calculating the characteristic value of the first attribute and the characteristic value of the second attribute according to a preset combination method to obtain a combined characteristic value;
[0136] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, and the continuous growth eigenvalue of each attribute are taken as the first feature set, including:
[0137] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, the continuous growth eigenvalue of each attribute and the combined eigenvalue are taken as the first feature set.
[0138] Specifically, before taking the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, and the continuous growth eigenvalue of each attribute as the first feature set, the eigenvalue of the first attribute and the eigenvalue of the second attribute can also be determined in the log data of the disk corresponding to each time interval, wherein the first attribute can be understood as the first important attribute in the log data, and the second attribute can be understood as the second important attribute in the log data, and the importance of the attribute can be preset. In the embodiment of the present application, only the eigenvalue of the first important attribute and the eigenvalue of the second important attribute can be determined in order according to the importance. Then, the eigenvalue of the first attribute and the eigenvalue of the second attribute can be combined and calculated according to the preset combination method to obtain the combined eigenvalue.
[0139] It is understandable that the method of considering the combined feature value can take into account multiple features at the same time to improve the stability between the features, and the number of important attributes may not be limited in the embodiment of the present application.
[0140] Furthermore, after obtaining the combined feature value in the log data of the disk corresponding to each time interval, the weight feature value of each attribute, the difference feature value of each attribute, the maximum feature value of each attribute, the minimum feature value of each attribute, the mean feature value of each attribute, the variance feature value of each attribute, the slope feature value of each attribute, the continuous growth feature value of each attribute, and the combined feature value can be used as the first feature set. Among them, the combined feature value in the log data of the disk corresponding to each time interval can be understood as the combined feature in the log data of the disk, so as to further avoid the problems of large noise and little valuable information in the log data.
[0141] It should also be noted that the determined first feature set may introduce linearly related features and some useless features, which may have a negative impact on the disk failure prediction model. Based on this, in the embodiment of the present application, the first feature set can be selected based on backward search and Wrapper feature evaluation. The specific steps are as follows: given a feature set {a1, a2, a3...ad}, remove one feature from the initial selected set to form a candidate subset containing (d-1) features. Among the (d-1) (d-1)-dimensional candidate feature subsets, select the optimal feature subset according to the prediction performance index of the training model as the selected feature set for the next round, until the optimal candidate feature subset of this round is worse than that of the previous round. At this time, the time complexity increases from 0( ) is reduced to 0( ).
[0142] Step 106: cluster the first feature set based on a density clustering algorithm, and perform undersampling on the first feature set after the clustering process to obtain a second feature set.
[0143] Since most disks are in normal working condition in most cases, the number of disk failure log records is much smaller than the number of normal log records, and the disk log data has the problem of extreme sample imbalance. One of the classic solutions to deal with the sample imbalance problem is undersampling, the main idea of which is to randomly extract some samples from the majority class samples and merge them with the minority class samples to form a relatively balanced set. Since the traditional method uses a random method to extract majority class samples, it may cause a large number of samples of one class to be deleted, while all samples of another class are retained, and it is impossible to cover all representative samples, resulting in the loss of important information. Based on this, this application proposes an undersampling method based on density clustering.
[0144] Specifically, after obtaining the first feature set, the eigenvalues corresponding to normal disks in the first feature set can be clustered based on a density clustering algorithm to obtain at least two clusters, each of which may include eigenvalues corresponding to at least two normal disks, and the eigenvalues in each cluster have similar characteristics.
[0145] Furthermore, all eigenvalues included in at least two clusters may be eliminated from the first eigenvalue set, and the set of all remaining eigenvalues may be used as a third eigenvalue set, and the eigenvalues in the third eigenvalue set may be eigenvalues corresponding to damaged disks.
[0146] Furthermore, at least two clusters and the third eigenvalue set can be merged according to a preset ratio, so that the ratio between the eigenvalues in each cluster and the eigenvalues in the combined third eigenvalue set is the preset ratio, thereby ensuring that the second feature set obtained after the combined processing can retain the original value information to the greatest extent.
[0147] See here Figure 2 The schematic diagram of the effect of obtaining the second feature set provided by the embodiment of the present application is shown. Figure 2 As shown, the disk feature data set (i.e., the first feature set) can be clustered to obtain n clusters, each of which can be represented by C1, C2, C3...Cn. Then, according to the preset ratio of 80%, eight eigenvalues are selected from each cluster and combined with any two eigenvalues in the third eigenvalue set, so that each cluster in the new disk feature data set includes eight original eigenvalues and two new eigenvalues.
[0148] Step 108: predict a feature set of the disk at the next moment according to the second feature set, and determine the fault information of the disk based on the feature set of the disk at the next moment.
[0149] The undersampling method based on density clustering can alleviate the imbalance problem of disk sample data to a certain extent, but it does not fundamentally solve the problem. In order to enable the disk failure prediction model to be applied to the real production environment and to accurately predict disk failures in the case of extreme imbalance between positive and negative samples, the prediction model is combined in the embodiment of the present application to effectively and accurately determine the disk failure information by predicting the feature set of the disk at the next moment.
[0150] Specifically, after obtaining the second feature set, the second feature set can be input into the trained first neural network to predict the feature set corresponding to the faulty disk. Figure 3 The diagram of the training effect of a first neural network provided in the embodiment of the present application is shown in FIG. Figure 3 As shown, the first neural network can be understood as a lightweight integrated learning model, which divides the training set into multiple subsets during the training process, and each subset is trained and verified separately to train multiple classifiers: C1, C2...Cn. Then, multiple classifiers can be voted according to weights, and preliminary prediction results can be obtained based on the determined weights, where the classifier with better results on the verification set has a greater voting weight.
[0151] Furthermore, after predicting the feature set corresponding to the faulty disk according to the first neural network, a sample feature set with the same number as the feature set corresponding to the faulty disk can be randomly selected from the sample set. The sample set can be understood as the feature set corresponding to the normal disk, and the feature set and the sample feature set corresponding to the faulty disk are divided and processed according to preset time intervals. For example, the feature set and the sample feature set corresponding to the faulty disk at time t0, the feature set and the sample feature set corresponding to the faulty disk at time t1, the feature set and the sample feature set corresponding to the faulty disk at time t2... until the feature set and the sample feature set corresponding to the faulty disk at time tn can be obtained respectively.
[0152] Furthermore, the feature set and sample feature set corresponding to the faulty disk at each moment can be encoded to remove redundant information, and the encoded feature set and sample feature set corresponding to the faulty disk at each moment can be input into the second neural network to predict the feature set of the disk at the next moment. Figure 4 FIG. 1 is a schematic diagram showing the training effect of a second neural network provided in an embodiment of the present application. Figure 4As shown, the feature set and the sample feature set corresponding to the faulty disk predicted by the first neural network can be combined and processed first, and the feature set and the sample feature set corresponding to the faulty disk can be divided into the feature set and the sample feature set corresponding to the faulty disk at different moments based on the time window. Then, the feature set and the sample feature set corresponding to the faulty disk at different moments can be encoded, and the feature set and the sample feature set corresponding to the faulty disk at different moments after encoding can be input into the second neural network, and the feature set of the disk at the next moment can be predicted by the timing neural unit of the second neural network.
[0153] Furthermore, after predicting the feature set of the disk at the next moment, the fault information of the disk can be determined based on the feature set of the disk at the next moment, wherein the fault information of the disk may include but is not limited to abnormal external vibration, abnormal temperature and humidity around the disk, and increased read and write error rate, etc., and after determining the fault information of the disk, the staff can be notified in time to handle it to avoid unnecessary losses.
[0154] In combination with one or more of the above-mentioned embodiments, on the one hand, the problem of large noise and little valuable information in disk log data is solved, and a feature construction method for disk data time series is designed, which mainly includes window features, statistical features, trend features and combination features, and then valuable time series features are extracted from disk log data;
[0155] On the other hand, the problem that traditional undersampling methods easily cause important information loss is solved. A disk data undersampling method based on density clustering is proposed. Before undersampling, the normal disk feature data is clustered and each cluster is undersampled, so as to ensure that the new data set can cover more types of disks and retain valuable information to the greatest extent.
[0156] On the other hand, it also solves the problem that disk failure prediction models are difficult to apply under extreme class imbalance. A two-stage disk failure prediction model is proposed. For unbalanced disk feature data, a lightweight ensemble learning model is first used to preliminarily predict the failed disk, and then combined with an equal number of normal disks into a new data set. The disk failure prediction model is then trained based on feature encoding and time series learning, thereby improving the prediction accuracy as much as possible and making full use of the disk space.
[0157] See also Figure 5 , Figure 5 A schematic diagram of the structure of a disk failure prediction device for processing extremely unbalanced data provided by an embodiment of the present application is shown.
[0158] like Figure 5As shown, the disk failure prediction device for processing extremely unbalanced data may include at least a log processing module 501, a feature extraction module 502, a feature processing module 503 and a model prediction module 504, wherein:
[0159] The log processing module 501 is used to obtain the log data of the disk and pre-process the log data of the disk;
[0160] The feature extraction module 502 is used to divide the log data of the disk after the preprocessing based on a preset time interval, and extract a first feature set from the log data of the disk after the division processing;
[0161] The feature processing module 503 is used to perform clustering processing on the first feature set based on a density clustering algorithm, and perform undersampling processing on the first feature set after the clustering processing to obtain a second feature set;
[0162] The model prediction module 504 is used to predict the feature set of the disk at the next moment according to the second feature set, and determine the fault information of the disk based on the feature set of the disk at the next moment.
[0163] In some possible embodiments, the log processing module includes:
[0164] Determine whether there is any missing data in the log data of the disk;
[0165] When it is determined that there are missing data in the log data of the disk, the missing data is interpolated by the mean value;
[0166] Normalize the log data of the disk after mean interpolation.
[0167] In some possible embodiments, the feature extraction module includes:
[0168] Determine the feature values belonging to the same attribute from the log data of the disk corresponding to each time interval, and calculate the weight feature value of each attribute according to the preset weight and each feature value belonging to the same attribute;
[0169] Determine the feature values belonging to the same attribute from the log data of the disk corresponding to any two consecutive time intervals, and perform difference calculation on the two feature values belonging to the same attribute to calculate the difference feature value of each attribute;
[0170] The weight feature value of each attribute and the difference feature value of each attribute are taken as the first feature set.
[0171] In some possible embodiments, the feature extraction module further includes:
[0172] Determine the maximum eigenvalue of each attribute and the minimum eigenvalue of each attribute from each eigenvalue belonging to the same attribute;
[0173] Calculate the mean of each eigenvalue belonging to the same attribute to obtain the mean eigenvalue of each attribute;
[0174] Calculate the variance of each eigenvalue belonging to the same attribute to obtain the variance eigenvalue of each attribute;
[0175] The weight feature value of each attribute and the difference feature value of each attribute are used as the first feature set, including:
[0176] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute are taken as the first feature set.
[0177] In some possible embodiments, the feature extraction module further includes:
[0178] Determine the characteristic values belonging to the same attribute from the log data of the disk corresponding to any two consecutive time intervals, and calculate the slope according to the coordinates corresponding to the two characteristic values belonging to the same attribute to obtain the slope characteristic value of each attribute;
[0179] Determine the number of eigenvalues whose values are in continuous growth among each eigenvalue belonging to the same attribute, and use the number of eigenvalues as the continuously growing eigenvalue of each attribute;
[0180] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute are used as the first feature set, including:
[0181] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute and the continuous growth eigenvalue of each attribute are taken as the first feature set.
[0182] In some possible embodiments, the feature extraction module further includes:
[0183] Determining a characteristic value of the first attribute and a characteristic value of the second attribute in log data of the disk corresponding to each time interval;
[0184] Combining and calculating the characteristic value of the first attribute and the characteristic value of the second attribute according to a preset combination method to obtain a combined characteristic value;
[0185] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, and the continuous growth eigenvalue of each attribute are taken as the first feature set, including:
[0186] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, the continuous growth eigenvalue of each attribute and the combined eigenvalue are taken as the first feature set.
[0187] In some possible embodiments, the feature processing module includes:
[0188] Performing clustering processing on the first feature set based on a density clustering algorithm to obtain at least two clusters; wherein each cluster includes at least two feature values;
[0189] Eliminate all feature values included in at least two clusters in the first feature set to obtain a third feature set;
[0190] At least two clusters and the third feature set are merged according to a preset ratio to obtain a second feature set.
[0191] In some possible embodiments, the model prediction module includes:
[0192] The second feature set is input into the trained first neural network to predict the feature set corresponding to the faulty disk;
[0193] Determine a sample feature set having the same number as the feature set corresponding to the faulty disk, and divide and process the feature set corresponding to the faulty disk and the sample feature set according to a preset time interval;
[0194] The feature set and sample feature set corresponding to the faulty disk after the division processing are encoded, and the feature set and sample feature set corresponding to the faulty disk after the encoding processing are input into the trained second neural network to predict the feature set of the disk at the next moment.
[0195] See also Figure 6 , Figure 6 A schematic diagram of the structure of another disk failure prediction device for processing extremely unbalanced data provided by an embodiment of the present application is shown.
[0196] like Figure 6As shown, the disk failure prediction device 600 for processing extremely unbalanced data may include at least one processor 601 , at least one network interface 604 , a user interface 603 , a memory 605 , and at least one communication bus 602 .
[0197] The communication bus 602 may be used to realize the connection and communication among the above-mentioned components.
[0198] The user interface 603 may include buttons, and the optional user interface may also include a standard wired interface or a wireless interface.
[0199] The network interface 604 may include, but is not limited to, a Bluetooth module, an NFC module, a Wi-Fi module, etc.
[0200] The processor 601 may include one or more processing cores. The processor 601 uses various interfaces and lines to connect various parts of the disk failure prediction device 600 that processes extremely unbalanced data, and executes various functions and processes data of the disk failure prediction device 600 that processes extremely unbalanced data by running or executing instructions, programs, code sets or instruction sets stored in the memory 605, and calling data stored in the memory 605. Optionally, the processor 601 can be implemented in at least one hardware form of DSP, FPGA, and PLA. The processor 601 can integrate one or a combination of CPU, GPU, and modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 601, but may be implemented separately through a chip.
[0201] The memory 605 may include a RAM or a ROM. Optionally, the memory 605 includes a non-transitory computer-readable medium. The memory 605 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 605 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 605 may optionally be at least one storage device located away from the aforementioned processor 601. As Figure 4 As shown, the memory 605 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a disk failure prediction application program for processing extremely unbalanced data.
[0202] Specifically, the processor 601 may be used to call a disk failure prediction application for processing extremely unbalanced data stored in the memory 605, and specifically perform the following operations:
[0203] Obtain the log data of the disk and preprocess the log data of the disk;
[0204] Performing segmentation processing on the log data of the disk after preprocessing based on a preset time interval, and extracting a first feature set from the log data of the disk after segmentation processing;
[0205] Performing clustering processing on the first feature set based on a density clustering algorithm, and performing undersampling processing on the first feature set after the clustering processing to obtain a second feature set;
[0206] A feature set of the disk at the next moment is predicted according to the second feature set, and fault information of the disk is determined based on the feature set of the disk at the next moment.
[0207] In some possible embodiments, the log data of the disk is preprocessed, including:
[0208] Determine whether there is any missing data in the log data of the disk;
[0209] When it is determined that there are missing data in the log data of the disk, the missing data is interpolated by the mean value;
[0210] Normalize the log data of the disk after mean interpolation.
[0211] In some possible embodiments, extracting a first feature set from log data of a disk after partitioning includes:
[0212] Determine the feature values belonging to the same attribute from the log data of the disk corresponding to each time interval, and calculate the weight feature value of each attribute according to the preset weight and each feature value belonging to the same attribute;
[0213] Determine the feature values belonging to the same attribute from the log data of the disk corresponding to any two consecutive time intervals, and perform difference calculation on the two feature values belonging to the same attribute to calculate the difference feature value of each attribute;
[0214] The weight feature value of each attribute and the difference feature value of each attribute are taken as the first feature set.
[0215] In some possible embodiments, before taking the weight feature value of each attribute and the difference feature value of each attribute as the first feature set, the method further includes:
[0216] Determine the maximum eigenvalue of each attribute and the minimum eigenvalue of each attribute from each eigenvalue belonging to the same attribute;
[0217] Calculate the mean of each eigenvalue belonging to the same attribute to obtain the mean eigenvalue of each attribute;
[0218] Calculate the variance of each eigenvalue belonging to the same attribute to obtain the variance eigenvalue of each attribute;
[0219] The weight feature value of each attribute and the difference feature value of each attribute are used as the first feature set, including:
[0220] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute are taken as the first feature set.
[0221] In some possible embodiments, before taking the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute as the first feature set, the method further includes:
[0222] Determine the characteristic values belonging to the same attribute from the log data of the disk corresponding to any two consecutive time intervals, and calculate the slope according to the coordinates corresponding to the two characteristic values belonging to the same attribute to obtain the slope characteristic value of each attribute;
[0223] Determine the number of eigenvalues whose values are in continuous growth among each eigenvalue belonging to the same attribute, and use the number of eigenvalues as the continuously growing eigenvalue of each attribute;
[0224] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, and the variance eigenvalue of each attribute are used as the first feature set, including:
[0225] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute and the continuous growth eigenvalue of each attribute are taken as the first feature set.
[0226] In some possible embodiments, before taking the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, and the continuous growth eigenvalue of each attribute as the first feature set, the method further includes:
[0227] Determining a characteristic value of the first attribute and a characteristic value of the second attribute in log data of the disk corresponding to each time interval;
[0228] Combining and calculating the characteristic value of the first attribute and the characteristic value of the second attribute according to a preset combination method to obtain a combined characteristic value;
[0229] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, and the continuous growth eigenvalue of each attribute are taken as the first feature set, including:
[0230] The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, the continuous growth eigenvalue of each attribute and the combined eigenvalue are taken as the first feature set.
[0231] In some possible embodiments, clustering is performed on the first feature set based on a density clustering algorithm, and undersampling is performed on the first feature set after clustering to obtain a second feature set, including:
[0232] Performing clustering processing on the first feature set based on a density clustering algorithm to obtain at least two clusters; wherein each cluster includes at least two feature values;
[0233] Eliminate all feature values included in at least two clusters in the first feature set to obtain a third feature set;
[0234] At least two clusters and the third feature set are merged according to a preset ratio to obtain a second feature set.
[0235] In some possible embodiments, predicting the feature set of the disk at the next moment according to the second feature set includes:
[0236] Inputting the second feature set into the trained first neural network to predict the feature set corresponding to the faulty disk;
[0237] Determine a sample feature set having the same number as the feature set corresponding to the faulty disk, and divide and process the feature set corresponding to the faulty disk and the sample feature set according to a preset time interval;
[0238] The feature set and sample feature set corresponding to the faulty disk after the division processing are encoded, and the feature set and sample feature set corresponding to the faulty disk after the encoding processing are input into the trained second neural network to predict the feature set of the disk at the next moment.
[0239] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.
[0240] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0241] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0242] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.
[0243] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0244] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0245] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.
[0246] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by entering a program to instruct related hardware, and the program may be stored in a computer-readable memory, which may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0247] The above are only exemplary embodiments of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.
Claims
1. A disk failure prediction method for processing extremely unbalanced data, characterized in that: include: Obtaining log data of a disk and preprocessing the log data of the disk; Performing segmentation processing on the log data of the disk after preprocessing based on a preset time interval, and extracting a first feature set from the log data of the disk after segmentation processing; Performing clustering processing on the first feature set based on a density clustering algorithm, and performing undersampling processing on the first feature set after the clustering processing to obtain a second feature set; Predicting a feature set of the disk at a next moment according to the second feature set, and determining fault information of the disk based on the feature set of the disk at the next moment; The step of extracting the first feature set from the log data of the disk after the partitioning process includes: Determining feature values belonging to the same attribute from the log data of the disk corresponding to each time interval, and calculating a weight feature value of each attribute according to a preset weight and each feature value belonging to the same attribute; Determine the characteristic values belonging to the same attribute from the log data of the disk corresponding to any two consecutive time intervals, and perform difference calculation on the two characteristic values belonging to the same attribute to calculate the difference characteristic value of each attribute; Taking the weight feature value of each attribute and the difference feature value of each attribute as the first feature set; The method of clustering the first feature set based on the density clustering algorithm and performing undersampling on the first feature set after the clustering process to obtain a second feature set includes: Performing clustering processing on the first feature set based on a density clustering algorithm to obtain at least two clusters; wherein each of the clusters includes at least two feature values; Eliminate all feature values included in the at least two clusters in the first feature set to obtain a third feature set; Merging the at least two clusters and the third feature set according to a preset ratio to obtain a second feature set; The step of predicting the feature set of the disk at the next moment according to the second feature set includes: Inputting the second feature set into the trained first neural network to predict the feature set corresponding to the faulty disk; determining a sample feature set having the same number as the feature set corresponding to the faulty disk, and dividing the feature set corresponding to the faulty disk and the sample feature set according to a preset time interval; The feature set corresponding to the faulty disk after the division processing and the sample feature set are encoded, and the feature set corresponding to the faulty disk after the encoding processing and the sample feature set are input into the trained second neural network to predict the feature set of the disk at the next moment.
2. The method according to claim 1, characterized in that The preprocessing of the log data of the disk includes: Determine whether there is missing data in the log data of the disk; When it is determined that there is missing data in the log data of the disk, performing mean interpolation processing on the missing data; The log data of the disk after the mean interpolation processing is normalized.
3. The method according to claim 1, characterized in that Before taking the weight feature value of each attribute and the difference feature value of each attribute as the first feature set, the method further includes: Determining the maximum eigenvalue of each attribute and the minimum eigenvalue of each attribute from each of the eigenvalues belonging to the same attribute; Calculate the mean of each eigenvalue belonging to the same attribute to obtain the mean eigenvalue of each attribute; Performing variance calculation on each of the eigenvalues belonging to the same attribute to obtain a variance eigenvalue of each attribute; The step of taking the weight feature value of each attribute and the difference feature value of each attribute as a first feature set includes: The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute and the variance eigenvalue of each attribute are taken as the first feature set.
4. The method according to claim 3, characterized in that Before taking the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute and the variance eigenvalue of each attribute as the first feature set, the method further includes: Determine the characteristic values belonging to the same attribute from the log data of the disk corresponding to any two consecutive time intervals, and calculate the slope according to the coordinates corresponding to the two characteristic values belonging to the same attribute to obtain the slope characteristic value of each attribute; Determine the number of eigenvalues whose values are in continuous growth among each of the eigenvalues belonging to the same attribute, and use the number of eigenvalues as the continuously growing eigenvalue of each attribute; The step of taking the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute and the variance eigenvalue of each attribute as a first feature set includes: The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute and the continuous growth eigenvalue of each attribute are taken as the first feature set.
5. The method according to claim 4, characterized in that Before taking the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, and the continuous growth eigenvalue of each attribute as the first feature set, the method further includes: Determining a characteristic value of a first attribute and a characteristic value of a second attribute in log data of the disk corresponding to each time interval; Combining and calculating the characteristic value of the first attribute and the characteristic value of the second attribute according to a preset combination method to obtain a combined characteristic value; The step of taking the weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, and the continuous growth eigenvalue of each attribute as a first feature set includes: The weight eigenvalue of each attribute, the difference eigenvalue of each attribute, the maximum eigenvalue of each attribute, the minimum eigenvalue of each attribute, the mean eigenvalue of each attribute, the variance eigenvalue of each attribute, the slope eigenvalue of each attribute, the continuous growth eigenvalue of each attribute and the combined eigenvalue are taken as the first feature set.
6. A disk failure prediction device for processing extremely unbalanced data, characterized in that: include: A log processing module, used for obtaining log data of the disk and preprocessing the log data of the disk; A feature extraction module, configured to perform segmentation processing on the log data of the disk after preprocessing based on a preset time interval, and extract a first feature set from the log data of the disk after segmentation processing; A feature processing module, configured to perform clustering processing on the first feature set based on a density clustering algorithm, and perform undersampling processing on the first feature set after the clustering processing to obtain a second feature set; A model prediction module, used to predict a feature set of the disk at a next moment according to the second feature set, and determine the fault information of the disk based on the feature set of the disk at the next moment; The step of extracting the first feature set from the log data of the disk after the partitioning process includes: Determining feature values belonging to the same attribute from the log data of the disk corresponding to each time interval, and calculating a weight feature value of each attribute according to a preset weight and each feature value belonging to the same attribute; Determine the characteristic values belonging to the same attribute from the log data of the disk corresponding to any two consecutive time intervals, and perform difference calculation on the two characteristic values belonging to the same attribute to calculate the difference characteristic value of each attribute; Taking the weight feature value of each attribute and the difference feature value of each attribute as the first feature set; The method of clustering the first feature set based on the density clustering algorithm and performing undersampling on the first feature set after the clustering process to obtain a second feature set includes: Performing clustering processing on the first feature set based on a density clustering algorithm to obtain at least two clusters; wherein each of the clusters includes at least two feature values; Eliminate all feature values included in the at least two clusters in the first feature set to obtain a third feature set; Merging the at least two clusters and the third feature set according to a preset ratio to obtain a second feature set; The step of predicting the feature set of the disk at the next moment according to the second feature set includes: Inputting the second feature set into the trained first neural network to predict the feature set corresponding to the faulty disk; determining a sample feature set having the same number as the feature set corresponding to the faulty disk, and dividing the feature set corresponding to the faulty disk and the sample feature set according to a preset time interval; The feature set corresponding to the faulty disk after the division processing and the sample feature set are encoded, and the feature set corresponding to the faulty disk after the encoding processing and the sample feature set are input into the trained second neural network to predict the feature set of the disk at the next moment.
7. A disk failure prediction device for processing extremely unbalanced data, characterized in that: including a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the steps of the method according to any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the steps of the method according to any one of claims 1 to 5.
