Disk fault prediction method and device for realizing cloud storage environment based on large model

Abstract

The embodiment of the invention provides a disk fault prediction method and device for realizing a cloud storage environment based on a large model, and the method comprises the steps: constructing a prediction model which comprises the prediction of a corresponding relation between reference feature data of a disk and a reference prediction result, obtaining the operation state data of a to-be-predicted disk in the cloud storage environment, and carrying out the prediction of the operation state data of the to-be-predicted disk in the cloud storage environment; and performing feature extraction on the operation state data to obtain feature data, obtaining a prediction result corresponding to the feature data according to the feature data and the prediction model, and generating a protection measure according to the prediction result to protect the to-be-predicted disk. By constructing the prediction model and inputting the feature data of the to-be-predicted disk into the prediction model, the prediction result can be obtained, and the protection measure is generated through the prediction result, so that prevention can be performed in advance when the disk breaks down, the security of the disk is protected, and data in the disk is prevented from being lost.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a disk failure prediction method and device for a cloud storage environment based on a large model. Background Art

[0002] With the rapid development of information technology, cloud storage has become the core infrastructure for modern data storage and access, providing flexible, scalable, and low-cost storage solutions for individual users, enterprises, and organizations. However, with the explosive growth of data volume and the increasing demand for data access speed, hardware failures in cloud storage environments, especially disk failures, have become a major challenge that seriously affects service quality and data security. Disk failures not only lead to data loss and service interruptions, but also increase recovery costs and reduce user experience, which in turn affects the business continuity and reputation of the enterprise.

[0003] Traditional disk failure detection methods mostly rely on hardware self-checks (such as SMART information monitoring) and regular manual inspections. However, hardware self-checks (such as SMART) and regular manual inspections usually report problems only after a failure occurs, resulting in failure to prevent data loss or system crashes in a timely manner, and it is difficult to take preventive measures before a failure occurs. Summary of the invention

[0004] The present invention provides a disk failure prediction method and device for a cloud storage environment based on a large model, which solves the problem in the prior art that hardware self-check methods (such as SMART) and regular manual inspections usually report problems only after a failure occurs, resulting in failure to prevent data loss or system crashes in a timely manner.

[0005] According to one aspect of the present invention, a disk failure prediction method for a cloud storage environment based on a large model is provided, comprising:

[0006] Based on the random forest algorithm and the deep learning model, a prediction model is constructed; wherein the prediction model includes a corresponding relationship between reference feature data of the predicted disk and a reference prediction result;

[0007] Obtaining the operating status data of the disk to be predicted in the cloud storage environment;

[0008] Extracting features from the operating status data to obtain feature data;

[0009] According to the feature data and the prediction model, obtaining a prediction result corresponding to the feature data;

[0010] According to the prediction result, protective measures are generated to protect the disk to be predicted.

[0011] In one embodiment, after obtaining the operating status data of the disk to be predicted in the cloud storage environment, the disk failure prediction method for the cloud storage environment based on the large model further includes:

[0012] Removing abnormal data from the operating status data;

[0013] Performing filling processing on the removed running status data to obtain filled running status data;

[0014] Performing format conversion on the filled running status data to obtain converted running status data;

[0015] Normalizing the converted running status data to obtain normalized running status data;

[0016] The step of extracting features from the operating status data to obtain feature data includes:

[0017] Feature extraction is performed on the normalized operating status data to obtain feature data.

[0018] In one embodiment, removing abnormal data from the operating status data includes:

[0019] Determining a first quartile in the operating status data; wherein the first quartile represents the 25th percentile in the operating status data;

[0020] Determining a third quartile in the operating status data; wherein the third quartile represents the 75th percentile in the operating status data;

[0021] Calculate the interquartile range according to the first quartile and the third quartile;

[0022] According to the interquartile range, determining an abnormal upper threshold and an abnormal lower threshold;

[0023] Determining abnormal data in the operating status data according to the abnormal upper limit threshold and the abnormal lower limit threshold;

[0024] The abnormal data is removed.

[0025] In one embodiment, the filling process is performed on the removed running status data to obtain the filled running status data, which includes:

[0026] identifying missing values ​​in the removed operating status data;

[0027] Determining a distribution state of the operating state data;

[0028] If the distribution state indicates that the operating state data is linearly distributed within a preset time period, the missing value is filled with the mode.

[0029] In one embodiment, the disk failure prediction method for a cloud storage environment based on a large model further includes:

[0030] If the distribution state indicates that the operating state data is not distributed in a straight line within a preset time period, the missing value is filled with the mean value.

[0031] In one embodiment, the format conversion of the filled running status data to obtain converted running status data includes:

[0032] Determining the category data in the filled running status data;

[0033] The categorical data is converted into numerical data to obtain converted operating status data.

[0034] In one embodiment, extracting features from the operating status data to obtain feature data includes:

[0035] Determining time-continuous time series data in the operating status data;

[0036] Cutting the time series data into sliding windows of preset length;

[0037] Feature data is extracted from the sliding window.

[0038] In one embodiment, constructing a prediction model based on a random forest algorithm and a deep learning model includes:

[0039] Obtain reference operating status data of multiple predicted disks;

[0040] Based on the random forest algorithm and the reference operating status data, a random forest model is constructed;

[0041] extracting reference feature data from the random forest model;

[0042] Inputting the reference feature data into the LTSM model in the deep learning model to obtain the failure probability corresponding to the reference feature data; wherein the reference prediction result includes the failure probability;

[0043] A prediction model is constructed according to the reference feature data of the plurality of predicted disks and the plurality of reference prediction results.

[0044] In one embodiment, after constructing a prediction model based on the reference feature data of the plurality of predicted disks and the plurality of reference prediction results, the disk failure prediction method for a cloud storage environment based on the large model further includes:

[0045] Dividing the reference feature data into a preset number of subsets; wherein each subset includes normal features and abnormal features;

[0046] Determining the verification rounds corresponding to the preset number;

[0047] Based on the validation round, each subset is used as a validation set;

[0048] According to the normal features and abnormal features in the validation set, the AUC-ROC value of the prediction model in each validation round is evaluated;

[0049] The prediction model whose AUC-ROC value is greater than a preset AUC-ROC threshold is selected as the target model; wherein the AUC-ROC value includes at least two decimal places, and the values ​​of at least two decimal places are greater than a preset positive integer, and the value of the digit before the decimal point of the AUC-ROC value is 0.

[0050] According to another aspect of the present invention, there is provided a disk failure prediction device for a cloud storage environment based on a large model, comprising:

[0051] A construction module is used to construct a prediction model based on a random forest algorithm and a deep learning model; wherein the prediction model includes a corresponding relationship between reference feature data of the predicted disk and a reference prediction result;

[0052] An acquisition module, used for acquiring the operation status data of the disk to be predicted in the cloud storage environment;

[0053] An extraction module, used for performing feature extraction on the operating status data to obtain feature data;

[0054] A prediction module, used to obtain a prediction result corresponding to the feature data according to the feature data and the prediction model;

[0055] A generation module is used to generate protection measures according to the prediction result to protect the disk to be predicted.

[0056] The embodiment of the present invention provides a disk failure prediction method and device for a cloud storage environment based on a large model, including: constructing a prediction model, wherein the prediction model includes a corresponding relationship between reference feature data of a predicted disk and a reference prediction result, obtaining operation status data of the disk to be predicted in the cloud storage environment, performing feature extraction on the operation status data to obtain feature data, obtaining a prediction result corresponding to the feature data according to the feature data and the prediction model, and generating protective measures according to the prediction result to protect the disk to be predicted. By constructing a prediction model and inputting the feature data of the disk to be predicted into the prediction model, the prediction result can be obtained, and the protective measures are generated by the prediction result, so that prevention can be carried out in advance when a disk failure occurs, thereby protecting the security of the disk and preventing the loss of data in the disk. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flowchart of a disk failure prediction method for a cloud storage environment based on a large model provided by an exemplary embodiment of the present invention.

[0058] Figure 2 It is a flowchart of a disk failure prediction method for a cloud storage environment based on a large model provided by another exemplary embodiment of the present invention.

[0059] Figure 3 It is a structural diagram of a disk failure prediction device for a cloud storage environment based on a large model provided by an exemplary embodiment of the present invention.

[0060] Figure 4 It is a structural diagram of a disk failure prediction device for a cloud storage environment based on a large model provided by another exemplary embodiment of the present invention.

[0061] Figure 5 is a structural diagram of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0062] Figure 1 FIG. 1 is a flow chart of a disk failure prediction method for a cloud storage environment based on a large model provided by an exemplary embodiment of the present invention. Figure 1 As shown, the disk failure prediction method for cloud storage environment based on the large model includes:

[0063] Step 110: construct a prediction model based on the random forest algorithm and the deep learning model, wherein the prediction model includes a correspondence between reference feature data of the predicted disk and a reference prediction result.

[0064] In an embodiment of the present invention, a disk failure prediction model constructed based on a random forest algorithm and a deep learning model is intended to achieve efficient fault warning by analyzing the corresponding relationship between the reference feature data of the disk (such as SMART attributes, historical read and write loads, temperature changes, etc.) and the reference prediction results (i.e., the probability of failure or prediction label). First, the random forest algorithm uses its powerful feature selection ability to dig out features closely related to the health status of the disk, and improves the accuracy and robustness of the prediction through integrated learning. At the same time, the deep learning model extracts deeper relationships from nonlinear and complex features through a multi-layer neural network structure, further improving the prediction ability. To ensure the effectiveness of the model, methods such as cross-validation and hyperparameter tuning are used to optimize the model performance. Finally, through visualization and feature importance analysis, the influence of each feature in the prediction results is clarified, thereby establishing a complete set of fault prediction systems, so that the system can monitor the disk status in real time and give early warnings, reducing the risk of data loss. The large model in this application refers to the prediction model.

[0065] Step 120: Obtain the operating status data of the disk to be predicted in the cloud storage environment.

[0066] In an embodiment of the present invention, the present invention can collect the operating status data of the disk to be predicted on the storage node in the cloud storage environment, wherein the operating status data may include SMART (Self-Monitoring, Analysis and Reporting Technology) self-test information, I / O performance indicators, temperature records, disk load conditions, power usage cycles, and detailed system log data. SMART (Self-Monitoring, Analysis and Reporting Technology) self-test information may specifically include disk rotation speed, seek time, read and write error rate, start / stop cycle times, internal read and write errors, etc. I / O performance indicators specifically include throughput, latency, queue depth, etc. The above data is obtained through a written script or API, and then aggregated to a central data warehouse. Through extensive and in-depth data capture, detailed basic information is provided for subsequent fault prediction.

[0067] Step 130: Extract features from the operating status data to obtain feature data.

[0068] In an embodiment of the present invention, based on the time window and SMART (Self-Monitoring, Analysis and Reporting Technology) self-test information, the change rate and slope of the SMART parameters are calculated to reflect the deterioration rate of the parameters over time. The frequency, intensity (such as IOPS), delay distribution, etc. of I / O operations, as well as long intervals without operations are extracted from the I / O performance indicators as indicators of idle periods. The cross-correlation between temperature fluctuations and I / O activities is analyzed to determine whether there is a temperature-sensitive failure mode. By effectively refining these features, the working status of the disk and the potential failure risk can be more accurately portrayed.

[0069] Step 140: Obtain prediction results corresponding to the feature data according to the feature data and the prediction model.

[0070] In an embodiment of the present invention, after the characteristic data is input into the model, the model will output the failure probability, remaining service life or other reliability evaluation indicators of the disk to be predicted in a specific future time period, providing real-time and accurate judgment basis for failure prediction.

[0071] Step 150: Generate protection measures based on the prediction results to protect the disk to be predicted.

[0072] In an embodiment of the present invention, based on the prediction results, the system will automatically link the intelligent decision-making module to trigger a series of preventive maintenance strategies to ensure service continuity and data integrity. First, based on the prediction results, the system can automatically migrate data to healthy disks in advance to avoid data loss due to failures. In addition, the system will dynamically adjust the distribution of data in the cluster, optimize load balancing, reduce the pressure on high-risk disks, and reduce their failure probability. At the same time, intelligently schedule resources to prioritize computing and storage tasks to healthy hardware to ensure that the performance of critical applications is not affected. In addition, the system will arrange hardware replacement in a timely manner based on health monitoring data and feedback from the prediction model to ensure that all components are running in the best condition. This automated preventive maintenance strategy enables the system to take proactive action before a failure occurs, which not only improves the overall operation and maintenance efficiency, but also significantly reduces the risk of unexpected downtime and data loss, providing users with a more reliable service experience.

[0073] As the system continues to run, more and more disk failure cases are accumulated, and the prediction model will be regularly iterated and optimized to maintain its leading and adaptable prediction capabilities. By introducing hyperparameter tuning, the parameters of the model will be fine-tuned according to the new data set to achieve the best learning effect. In addition, through feature importance screening, the system will identify features that have a significant impact on fault prediction and remove redundant and noisy features, thereby improving the interpretability and training efficiency of the model. At the same time, using ensemble learning strategies such as random forests and gradient boosting trees, the prediction results of multiple models are integrated to further enhance the accuracy and robustness of the overall prediction. The combination of these advanced strategies enables the model to flexibly respond to the ever-changing challenges in the cloud storage environment, such as the surge in data volume, the diversification of user behavior, and the evolution of hardware configuration, thereby achieving more efficient fault warning and intelligent decision-making, ensuring the stability and reliability of the system. Through this dynamic optimization process, the prediction model can not only adapt to the current operating status, but also continue to learn and evolve, providing stronger support for future operation and maintenance decisions.

[0074] The embodiment of the present invention provides a disk failure prediction method for a cloud storage environment based on a large model, including: constructing a prediction model, wherein the prediction model includes a corresponding relationship between reference feature data of a predicted disk and a reference prediction result, obtaining operation status data of the disk to be predicted in the cloud storage environment, extracting features of the operation status data to obtain feature data, obtaining prediction results corresponding to the feature data according to the feature data and the prediction model, and generating protective measures according to the prediction results to protect the disk to be predicted. By constructing a prediction model and inputting the feature data of the disk to be predicted into the prediction model, the prediction results can be obtained, and protective measures are generated by the prediction results, so that prevention can be carried out in advance when a disk failure occurs, thereby protecting the security of the disk and preventing data loss in the disk.

[0075] Figure 2 FIG. 1 is a flow chart of a disk failure prediction method for a cloud storage environment based on a large model provided by another exemplary embodiment of the present invention. Figure 2 As shown, after step 120, the disk failure prediction method for a cloud storage environment based on a large model may further include:

[0076] Step 160: Remove abnormal data from the operating status data.

[0077] In the embodiment of the present invention, removing abnormal data in the operating status data is a key step to ensure the accuracy and reliability of the prediction model. Abnormal data generally refers to values ​​in the data set that are significantly different from the normal mode. These values ​​may be caused by factors such as sensor failure, data input errors, and environmental interference.

[0078] Specifically, step 160 may include the following steps:

[0079] Step 161: Determine the first quartile in the running status data; wherein the first quartile represents the 25th percentile in the running status data;

[0080] In an embodiment of the present invention, a first quartile in the operating status data is determined, and the first quartile is the 25th percentile in the operating status data.

[0081] Step 162: Determine the third quartile in the running status data; wherein the third quartile represents the 75th percentile in the running status data;

[0082] In an embodiment of the present invention, a third quartile in the operating status data is determined, and the third quartile is the 75th percentile in the operating status data.

[0083] Step 163: Calculate the interquartile range according to the first quartile and the third quartile;

[0084] In the embodiment of the present invention, the difference between the first quartile and the third quartile is calculated as the interquartile range.

[0085] Step 164: Determine an upper abnormal threshold and a lower abnormal threshold according to the interquartile range;

[0086] In the embodiment of the present invention, the abnormal upper threshold and the abnormal lower threshold can be calculated by the interquartile range. Specifically, the calculation formula of the abnormal lower threshold is: the lower threshold = Q1-1.5×IQR, where Q1 represents the first quartile and IQR represents the interquartile range. The calculation formula of the abnormal upper threshold is: the upper threshold = Q3+1.5×IQR, where Q3 represents the third quartile and IQR represents the interquartile range.

[0087] Step 165: determining abnormal data in the operating status data according to the abnormal upper threshold and the abnormal lower threshold;

[0088] In the embodiment of the present invention, data in the running status data that is smaller than an abnormal lower threshold or larger than an abnormal upper threshold is determined as abnormal data.

[0089] Step 166: Remove abnormal data.

[0090] In addition, you can also use data that is approximately normally distributed, and use the 3σ principle to remove values ​​that are beyond ±3 standard deviations from the mean. Specifically, calculate the mean and standard deviation of the operating status data, where the standard deviation is denoted by σ. Based on the mean and standard deviation, calculate the upper limit and lower limit, where the upper limit is calculated as: μ+3σ, where μ represents the mean and σ represents the standard deviation. The lower limit is calculated as: μ-3σ, where μ represents the mean and σ represents the standard deviation. Determine that data in the operating status data that is less than the lower limit or greater than the upper limit is abnormal data.

[0091] Among them, step 130 may include:

[0092] Step 131: extracting features from the normalized running status data to obtain feature data.

[0093] Step 170: Perform filling processing on the removed running status data to obtain filled running status data.

[0094] In the embodiment of the present invention, in the actual data collection process, especially in the operation status monitoring or time series data analysis, the data may be affected by noise, measurement error or other factors, resulting in missing values ​​or outliers. After removing the outliers, there may be gaps in the data set, affecting the subsequent data analysis and modeling. Therefore, it is necessary to reasonably fill these missing values.

[0095] Specifically, step 170 may include the following steps:

[0096] Step 171: Identify missing values ​​in the removed operating status data;

[0097] In an embodiment of the present invention, in the running status data after removing the outliers, the process of identifying missing values ​​usually involves several key steps. First, by checking the basic information of the data set, the data type and sample size of each feature (column) are determined to understand the overall structure of the data. Then, the number and proportion of missing values ​​in each feature are calculated using descriptive statistics, which can be easily implemented by programming languages ​​(such as Python's Pandas library) to generate a missing value report. At the same time, a visualization tool (such as a heat map or a missing value matrix) is used to intuitively display the distribution of missing values ​​in the data to help identify the pattern and potential relationship of missing values. In addition, analyzing whether the distribution of missing values ​​is systematic, that is, whether the missing of certain features is related to other features, can provide a basis for the selection of subsequent filling methods. Through these steps, the situation of missing values ​​in the data can be fully understood, laying the foundation for subsequent data cleaning and filling processing.

[0098] Step 172: Determine the distribution status of the operation status data;

[0099] In an embodiment of the present invention, determining the distribution state of the operating status data is an important step in data analysis, which helps to understand the characteristics and potential patterns of the data. First, descriptive statistical methods (such as mean, median, standard deviation and quartiles) can be used to summarize the central tendency and dispersion of the data. Next, visual graphics such as histograms and box plots are drawn to intuitively present the distribution of the data, including the skewness (left or right skewness), kurtosis (flat or sharp) of the data, and whether there are outliers. Through these visualizations, analysts can identify the distribution form of the data, such as whether it conforms to a normal distribution or presents a multimodal distribution. In addition, the assumptions of data distribution can be further verified using statistical tests (such as the Shapiro-Wilk test or the Kolmogorov-Smirnov test).

[0100] Step 173: If the distribution state indicates that the operating state data is linearly distributed within a preset time period, the missing values ​​are filled with the mode.

[0101] In an embodiment of the present invention, if the distribution state indicates that the running status data is linearly distributed within a preset time period, that is, the running status data has no upward or downward trend within the preset time period, the missing value is filled with the mean, or if the distribution state indicates that the number of repeated values ​​of the running status data is greater than the first number, the missing value is filled with the mode. Mode is a concept in statistics, which refers to the value with the highest frequency in a set of data. Simply put, the mode is the most common value in a data set. It can be used to describe the central tendency of data, and is particularly useful in the analysis of categorical data or discrete data.

[0102] Optionally, step 170 may further include the following steps:

[0103] Step 174: If the distribution state indicates that the running state data is not distributed in a straight line within the preset time period, the missing value is filled with the mean value.

[0104] In an embodiment of the present invention, if the distribution state indicates that the running status data is not linearly distributed within a preset time period, or the distribution state indicates that the number of repeated values ​​of the running status data is less than the first number, the missing value is filled with the mean. The mean is an important concept in statistics, usually referring to the arithmetic mean of a set of data. It is calculated by adding all data values ​​and then dividing by the number of data values. The mean is usually used to describe the central tendency of the data and reflect the overall level of the data.

[0105] Optionally, when dealing with missing values, the K-nearest neighbor prediction model (KNN) is an effective method to fill missing values, especially when the proportion of missing data is small. KNN predicts missing values ​​by calculating the similarity of other samples (usually based on a distance metric such as Euclidean distance). For example, for a sample with a missing feature, KNN will find its K closest neighbors in the feature space and perform a weighted average based on the known values ​​of these neighbors to fill in the missing value. This method takes advantage of the local structure of the data and can provide reasonable missing value estimates while maintaining the distribution characteristics of the data.

[0106] However, for features with too high a proportion of missing values, if these features have little impact on the results in the analysis, considering ignoring them directly can reduce the complexity of the analysis and potential noise interference. In this case, deleting records with missing values ​​may be a simple and effective option, especially when the data set is large, and doing so will not significantly affect the overall analysis results. By weighing the proportion of missing values ​​and the importance of features, analysts can make more reasonable decisions, thereby improving the accuracy and effectiveness of data analysis.

[0107] Step 180: Perform format conversion on the filled running status data to obtain converted running status data.

[0108] In an embodiment of the present invention, unifying data formats and standards is a key step in data preprocessing, which aims to improve the availability of data and the consistency of analysis. First, different data sources often use different formats, such as the representation of date and time (such as YYYY-MM-DD and DD / MM / YYYY), which can cause confusion and errors when merging or analyzing data. Therefore, it is very important to unify the date and time format to ensure that all data can be processed under the same standard. In addition, unit standardization is also necessary, such as unifying temperature into degrees Celsius or degrees Fahrenheit, length into meters or feet, etc., so as to avoid analysis bias caused by inconsistent units.

[0109] When processing time series data, it is also important to ensure that the data is aligned at the same frequency and time interval. For example, when merging multiple time series, they need to be aggregated to the same time point, such as by hour, day, or week. This not only helps eliminate the interference caused by time inconsistency, but also makes subsequent trend analysis and forecasting models more accurate.

[0110] Optionally, step 180 may include the following steps:

[0111] Step 181: Determine the category data in the filled running status data;

[0112] In an embodiment of the present invention, in the filled operating status data, categorical data refers to information expressed in a classified form, which usually reflects the state, characteristics or classification attributes of the system rather than the numerical value. The step of determining categorical data includes identifying all non-numerical variables in the data set, such as fault type, equipment status (such as "normal", "warning", "fault"), operation mode (such as "automatic", "manual"), disk model, etc. For these data, label encoding or unique hot encoding and other techniques are usually used for processing, so that this information can be effectively used in subsequent analysis and modeling. The existence of categorical data provides important context for data analysis, which can help identify performance changes, failure modes, and operating efficiency of the system under different conditions.

[0113] Step 182: Convert the categorical data into numerical data to obtain converted operating status data.

[0114] In an embodiment of the present invention, converting categorical data into numerical data is an important step in data preprocessing, especially when building a machine learning model. Common methods include label encoding and one-hot encoding. Label encoding is suitable for ordered categorical data by assigning a unique integer value to each category. For example, "normal", "warning" and "fault" are encoded as 0, 1 and 2 respectively. One-hot encoding creates a new binary feature for each category. For each class, if the sample belongs to the category, the feature value is 1, otherwise it is 0. This method is particularly suitable for unordered categorical data, such as equipment type or operation mode. Through this conversion, the categorical information in the operating status data can be effectively used by the machine learning algorithm in the form of a numerical value, thereby improving the performance and accuracy of the model. It should be noted that choosing an appropriate encoding method is crucial to maintaining the original information and features of the data. For example, using label encoding may introduce unnecessary sequential relationships, while one-hot encoding effectively eliminates this risk, but increases the dimension of the data.

[0115] Step 190: normalize the converted running status data to obtain normalized running status data.

[0116] In the embodiment of the present invention, the comparability of data from different sources is ensured, so the minimum-maximum scaling is adopted to map the value of the feature corresponding to the converted running status data to the interval [0,1], and the formula is:

[0117]

[0118] Among them, x represents a certain feature value in the original data, min(x) is the minimum value in the feature column, max(x) is the maximum value in the feature column, and x normis the normalized value. The converted running status data is normalized by the above formula to obtain the normalized running status data.

[0119] In one embodiment, step 130 may be specifically implemented as follows: determining time series data that is continuous in time in the operating status data; cutting the time series data into sliding windows of a preset length; and extracting feature data from the sliding windows.

[0120] In an embodiment of the present invention, cutting continuous time series data into sliding windows of fixed length according to analysis requirements is an effective feature extraction method, which aims to capture local patterns and trends in time series data. Each sliding window can cover observations over a period of time. By calculating these data, a variety of useful statistical features can be extracted, such as the average, maximum, minimum and standard deviation within the window, which can reflect the basic changes in the data during the time period. In addition, trends and periodic components can be extracted through more complex analysis methods, such as using the moving average method to determine long-term trends, or using Fourier transforms to identify periodic fluctuations.

[0121] In one embodiment, step 110 can be specifically implemented as follows: obtaining reference operating status data of multiple predicted disks; constructing a random forest model based on a random forest algorithm and the reference operating status data; extracting reference feature data from the random forest model; inputting the reference feature data into the LTSM model in the deep learning model to obtain the failure probability corresponding to the reference feature data; wherein the reference prediction result includes the failure probability; and constructing a prediction model based on the reference feature data of multiple predicted disks and multiple reference prediction results.

[0122] In an embodiment of the present invention, a random forest algorithm is used to construct the reference operating status data of multiple predicted disks into a random forest model. Specifically, the number of trees, the maximum depth, and the minimum number of sample splits are set, and the reference operating status data is filled into each node of the tree according to the attributes, labels, etc. of the reference operating status data, thereby forming a random forest model. According to the predicted content, the reference feature data of the reference operating status data is extracted from the random forest model. And the reference feature data is input into the LTSM model in the deep learning model, so that the failure probability corresponding to the reference feature data can be obtained. LSTM (Long Short-Term Memory) is a special type of recursive neural network (RNN) designed to process and predict time series data.

[0123] Optionally, the reference feature data can also be input into the convolutional neural network model to obtain the fault probability corresponding to the reference feature data. Convolutional Neural Network (CNN) is a deep learning model that is widely used in image processing, computer vision and other fields. CNN can automatically learn to extract features from images by simulating the processing method of the human visual system, and then perform tasks such as classification and detection.

[0124] In one embodiment, step 110 can be specifically implemented as follows: determining a preset number of validation rounds; based on the validation rounds, using each subset as a validation set; evaluating the AUC-ROC value of the prediction model in each validation round according to the normal features and abnormal features in the validation set; selecting a prediction model whose AUC-ROC value is greater than a preset AUC-ROC threshold as the target model; wherein the AUC-ROC value includes at least two decimal places, and the values ​​of at least two decimal places are greater than a preset positive integer, and the value of the digit before the decimal point of the AUC-ROC value is 0.

[0125] In an embodiment of the present invention, a 5-fold cross-validation method is used to evaluate the accuracy of the prediction model. Specifically, the data set is evenly divided into 5 parts, each part is used as a validation set in turn, and the remaining parts are used as training sets to perform model training and validation, wherein the number of validation rounds is the same as the number of data sets after segmentation, that is, the validation argument is 5. The prediction model is made to learn the potential laws of failure occurrence using historical cases of disk failures and their characteristics. In each validation phase, the AUC-ROC (Area Under the Receiver Operating Characteristic Curve) is used as the main evaluation index to evaluate the model's ability to distinguish between normal and faulty disk states. The AUC-ROC value corresponding to each validation round is set to include at least two decimal places, and the values ​​of at least two decimal places are greater than a preset positive integer, and the model corresponding to the value of the digit before the decimal point of the AUC-ROC value being 0 has better discrimination. That is, a model with an AUC-ROC value close to 1 is considered to have better discrimination. Based on the average AUC-ROC value of the model under all cross-validation rounds, the model configuration with the highest score is selected as the final model. That is to say, the trained model corresponding to the largest value in the AUC-ROC value is selected as the prediction model. AUC-ROC (Area Under the Receiver Operating Characteristic Curve) is an important indicator for evaluating the performance of classification models, especially in binary classification tasks. It combines the true positive rate (TPR) and false positive rate (FPR) of the model at different thresholds to provide a comprehensive performance evaluation.

[0126] When the real-time disk operation status data is input into the model, the model will output the failure probability, remaining service life or other reliability evaluation indicators of each disk in a specific time period in the future, providing real-time and accurate judgment basis for failure prediction.

[0127] Figure 3 FIG. 1 is a schematic diagram of a disk failure prediction device for a cloud storage environment based on a large model provided by an exemplary embodiment of the present invention. Figure 3 As shown, the disk failure prediction device 20 for implementing a cloud storage environment based on a large model includes:

[0128] A construction module 201 is used to construct a prediction model based on a random forest algorithm and a deep learning model; wherein the prediction model includes a correspondence between reference feature data of a predicted disk and a reference prediction result; an acquisition module 202 is used to obtain operating status data of a disk to be predicted in a cloud storage environment; an extraction module 203 is used to perform feature extraction on the operating status data to obtain feature data; a prediction module 204 is used to obtain a prediction result corresponding to the feature data based on the feature data and the prediction model; a generation module 205 is used to generate protection measures based on the prediction result to protect the disk to be predicted.

[0129] The embodiment of the present invention provides a disk failure prediction device for a cloud storage environment based on a large model, including: a construction module constructs a prediction model based on a random forest algorithm and a deep learning model, wherein the prediction model includes a corresponding relationship between reference feature data of the predicted disk and a reference prediction result, an acquisition module acquires the operating status data of the disk to be predicted in the cloud storage environment, an extraction module performs feature extraction on the operating status data to obtain feature data, a prediction module acquires the prediction result corresponding to the feature data according to the feature data and the prediction model, and a generation module generates protective measures according to the prediction result to protect the disk to be predicted. By constructing a prediction model and inputting the feature data of the disk to be predicted into the prediction model, the prediction result can be obtained, and the protective measures are generated by the prediction result, so that prevention can be carried out in advance when the disk fails, thereby protecting the security of the disk and preventing the loss of data in the disk.

[0130] Figure 4 FIG. 1 is a schematic diagram of a disk failure prediction device for a cloud storage environment based on a large model provided by another exemplary embodiment of the present invention. Figure 4As shown, after the acquisition module 202, the disk failure prediction device 20 for implementing a cloud storage environment based on a large model can be specifically configured as follows: removing abnormal data from the operating status data; performing filling processing on the removed operating status data to obtain filled operating status data; performing format conversion on the filled operating status data to obtain converted operating status data; and performing normalization processing on the converted operating status data to obtain normalized operating status data.

[0131] In one embodiment, a disk failure prediction device 20 for a cloud storage environment implemented based on a large model can be specifically configured as follows: determining the first quartile in the operating status data; wherein the first quartile represents the 25th percentile in the operating status data; determining the third quartile in the operating status data; wherein the third quartile represents the 75th percentile in the operating status data; calculating the interquartile range based on the first quartile and the third quartile; determining an abnormal upper threshold and an abnormal lower threshold based on the interquartile range; determining abnormal data in the operating status data based on the abnormal upper threshold and the abnormal lower threshold; removing abnormal data; the extraction module 203 can be specifically configured as follows: performing feature extraction on the normalized operating status data to obtain feature data.

[0132] In one embodiment, a disk failure prediction device 20 for a cloud storage environment based on a large model can be specifically configured as follows: identifying missing values ​​in the removed operating status data; determining the distribution status of the operating status data; if the distribution status indicates that the operating status data is linearly distributed within a preset time period, then filling the missing value with the mode.

[0133] In one embodiment, the disk failure prediction device 20 for cloud storage environment based on the large model can be specifically configured as follows: if the distribution state indicates that the operation state data is not linearly distributed within a preset time period, the missing value is filled with the mean value.

[0134] In one embodiment, the disk failure prediction device 20 for implementing a cloud storage environment based on a large model can be specifically configured as follows: determining categorical data in the populated operating status data; converting the categorical data into numerical data to obtain converted operating status data.

[0135] In one embodiment, the extraction module 203 may include: a determination unit 2031, used to determine time series data that is time-continuous in the operating status data; a cutting unit 2032, used to cut the time series data into sliding windows of a preset length; and an extraction subunit 2033, used to extract feature data from the sliding window.

[0136] In one embodiment, the construction module 201 may include:

[0137] The state unit 2011 is used to obtain reference operation state data of a plurality of predicted disks;

[0138] A random forest model unit 2012, used for building a random forest model based on a random forest algorithm and reference operating status data;

[0139] A reference unit 2013, used for extracting reference feature data from the random forest model;

[0140] An input unit 2014 is used to input the reference feature data into the LTSM model in the deep learning model to obtain the failure probability corresponding to the reference feature data; wherein the reference prediction result includes the failure probability;

[0141] The construction subunit 2015 is used to construct a prediction model according to the reference feature data of multiple predicted disks and multiple reference prediction results.

[0142] In one embodiment, after constructing subunit 2015, the disk failure prediction device 20 for implementing a cloud storage environment based on a large model can be specifically configured as follows: determining a preset number of verification rounds; based on the verification rounds, using each subset as a verification set; evaluating the AUC-ROC value of the prediction model in each verification round according to normal features and abnormal features in the verification set; selecting a prediction model whose AUC-ROC value is greater than a preset AUC-ROC threshold as a target model; wherein the AUC-ROC value includes at least two decimal places, and the values ​​of at least two decimal places are both greater than a preset positive integer, and the value of the digit before the decimal point of the AUC-ROC value is 0.

[0143] Figure 5 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0144] like Figure 5 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0145] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0146] The memory 12 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the disk failure prediction method for implementing a cloud storage environment based on a large model of the various embodiments of the present application described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0147] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0148] When the electronic device 10 is a stand-alone device, the input device 13 may be a communication network connector for receiving collected input signals from the first device and the second device.

[0149] In addition, the input device 13 may also include, for example, a keyboard, a mouse, etc.

[0150] The output device 14 can output various information to the outside, including the determined distance information, direction information, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0151] Of course, to simplify, Figure 5 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0152] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0153] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0154] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A disk failure prediction method for cloud storage environment based on a large model, characterized in that: include: Based on the random forest algorithm and the deep learning model, a prediction model is constructed; wherein the prediction model includes a corresponding relationship between reference feature data of the predicted disk and a reference prediction result; Obtaining the operating status data of the disk to be predicted in the cloud storage environment; Extracting features from the operating status data to obtain feature data; According to the feature data and the prediction model, obtaining a prediction result corresponding to the feature data; According to the prediction result, protective measures are generated to protect the disk to be predicted.

2. The disk failure prediction method for cloud storage environment based on a large model according to claim 1 is characterized in that: After obtaining the operation status data of the disk to be predicted in the cloud storage environment, the method further includes: Removing abnormal data from the operating status data; Performing filling processing on the removed running status data to obtain filled running status data; Performing format conversion on the filled running status data to obtain converted running status data; Normalizing the converted running status data to obtain normalized running status data; The step of extracting features from the operating status data to obtain feature data includes: Feature extraction is performed on the normalized operating status data to obtain feature data.

3. The disk failure prediction method for cloud storage environment based on a large model according to claim 2 is characterized in that: The removing of abnormal data in the running status data comprises: Determining a first quartile in the operating status data; wherein the first quartile represents the 25th percentile in the operating status data; Determining a third quartile in the operating status data; wherein the third quartile represents the 75th percentile in the operating status data; Calculate the interquartile range according to the first quartile and the third quartile; According to the interquartile range, determining an abnormal upper threshold and an abnormal lower threshold; Determining abnormal data in the operating status data according to the abnormal upper limit threshold and the abnormal lower limit threshold; The abnormal data is removed.

4. The disk failure prediction method for cloud storage environment based on a large model according to claim 2 is characterized in that: The step of performing filling processing on the removed running status data to obtain filled running status data includes: identifying missing values ​​in the removed operating status data; Determining a distribution state of the operating state data; If the distribution state indicates that the operating state data is linearly distributed within a preset time period, the missing value is filled with the mode.

5. The disk failure prediction method for cloud storage environment based on a large model according to claim 4 is characterized in that: Also includes: If the distribution state indicates that the operating state data is not distributed in a straight line within a preset time period, the missing value is filled with the mean value.

6. The disk failure prediction method for cloud storage environment based on a large model according to claim 2 is characterized in that: The format conversion of the filled running status data to obtain the converted running status data includes: Determining the category data in the filled running status data; The categorical data is converted into numerical data to obtain converted operating status data.

7. The disk failure prediction method for cloud storage environment based on a large model according to claim 1 is characterized in that: The extracting features of the running status data to obtain feature data includes: Determining time-continuous time series data in the operating status data; Cutting the time series data into sliding windows of preset length; Feature data is extracted from the sliding window.

8. The disk failure prediction method for cloud storage environment based on a large model according to claim 1 is characterized in that: The prediction model constructed based on the random forest algorithm and the deep learning model includes: Obtain reference operating status data of multiple predicted disks; Based on the random forest algorithm and the reference operating status data, a random forest model is constructed; extracting reference feature data from the random forest model; Inputting the reference feature data into the LTSM model in the deep learning model to obtain the failure probability corresponding to the reference feature data; wherein the reference prediction result includes the failure probability; A prediction model is constructed according to the reference feature data of the plurality of predicted disks and the plurality of reference prediction results.

9. The disk failure prediction method for cloud storage environment based on a large model according to claim 8 is characterized in that: After constructing the prediction model according to the reference feature data of the plurality of predicted disks and the plurality of reference prediction results, the method further includes: Dividing the reference feature data into a preset number of subsets; wherein each subset includes normal features and abnormal features; Determining the verification rounds corresponding to the preset number; Based on the validation round, each subset is used as a validation set; According to the normal features and abnormal features in the validation set, the AUC-ROC value of the prediction model in each validation round is evaluated; The prediction model whose AUC-ROC value is greater than a preset AUC-ROC threshold is selected as the target model; wherein the AUC-ROC value includes at least two decimal places, and the values ​​of at least two decimal places are greater than a preset positive integer, and the value of the digit before the decimal point of the AUC-ROC value is 0.

10. A disk failure prediction device for cloud storage environment based on a large model, characterized in that: include: A construction module is used to construct a prediction model based on a random forest algorithm and a deep learning model; wherein the prediction model includes a corresponding relationship between reference feature data of the predicted disk and a reference prediction result; An acquisition module, used for acquiring the operation status data of the disk to be predicted in the cloud storage environment; An extraction module, used for performing feature extraction on the operating status data to obtain feature data; A prediction module, used to obtain a prediction result corresponding to the feature data according to the feature data and the prediction model; A generation module is used to generate protection measures according to the prediction result to protect the disk to be predicted.

Images