Hard disk health degree assessment early warning method and device based on multi-model integration and medium

Abstract

The invention relates to a hard disk health degree assessment and early warning method and device based on multi-model integration and a medium, and the method comprises the following steps: obtaining SMART log files running in real time in each hard disk node in a service cluster, and carrying out the preprocessing; the preprocessed SMART log file is input into a large model to be processed, the health degree score of each hard disk node is output, and the large model is integrated through multiple models; and judging whether the health degree score of each hard disk node exceeds a dynamic health threshold, if so, outputting an early warning signal, and if not, not outputting the early warning signal. Compared with the prior art, the method has the advantages that the risk that backup cannot be carried out due to the fact that the hard disk breaks down is reduced, and the purposes of cost reduction and efficiency improvement are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of storage devices, and in particular to a hard disk health assessment and early warning method, device and medium based on multi-model integration. Background Art

[0002] With the development of network information technology, data plays an important role in it, and hard disks are equally important for data storage. If the health of hard disks can be judged by data before they fail, and they are managed uniformly, the risk of being unable to back up data due to hard disk failure can be effectively avoided.

[0003] At present, there are three main methods for identifying hard disk failures: based on hard disk communication failures, based on a single SMART log indicator, and based on the voltage and current of the hard disk itself. The following will analyze the problems existing in the current methods from these categories.

[0004] In the traditional method of judging the fault of the hard disk based on whether it is online, patent application CN116795610A discloses a hard disk fault detection method, system, device, computer equipment and storage medium, which obtains the in-place status of a first number of hard disks to be detected; determines whether the hard disk to be detected is in place according to the in-place status; obtains the connection status of the hard disk to be detected when the hard disk to be detected is in place, and determines whether the hard disk to be detected is faulty according to the connection status; and sends fault information when it is determined that the hard disk to be detected is a faulty hard disk. This method cannot accurately locate which hard disk has a connection fault for all hard disks, but cannot determine whether each hard disk itself has a fault.

[0005] In the method of diagnosing the hard disk status based on the SMART log, patent application CN113077835A discloses a hard disk fault detection method, device, equipment and readable storage medium. The method is based on the operation log of each server. If the number of occurrences of at least one preset indicator item is not less than the preset number threshold corresponding to the preset indicator item, the hard disk of the server node is used as the target hard disk. Among them, the preset number threshold corresponding to each preset indicator item is configured according to the probability of failure of the hard disk of the server node. Further, this method only needs to obtain the SMART log of the target hard disk, and detect whether the target hard disk fails based on the SMART log of the target hard disk. This method only monitors a single indicator of the SMART log, which cannot reflect the actual situation of the hard disk status, and there will be false alarms.

[0006] In the judgment method based on the hard disk hardware itself, patent application CN107590040A discloses a hard disk backplane and computer device, a hard disk fault detection method and a memory. The method uses a voltage sensor and a current sensor to detect the voltage and current of the corresponding hard disk power line to see if the current is normal. If it is abnormal, an alarm is issued. This enables accurate and efficient hard disk fault location. This method can detect a single hard disk, but cannot detect hard disks of each node of a large number of servers, and the operation is time-consuming and laborious.

[0007] In summary, the current hard disk fault detection method has the following problems: First, it determines whether a fault occurs in the hard disk based on whether the hard disk is communicating and has a life heartbeat, which only solves the communication fault but does not detect the hard disk fault itself; secondly, it makes a threshold judgment based on a single indicator in the SMART log file, and diagnoses whether the hard disk fault occurs by the number of occurrences. The single indicator cannot fully explain the status of the hard disk itself, and there is a risk of false alarm; thirdly, the voltage and current of the hard disk power line are judged through hardware settings, which is complicated to operate and cannot be judged on the hard disks of each node in the cluster server. At the same time, it is impossible to perform early warning processing on the status of the hard disk. Summary of the invention

[0008] The purpose of the present invention is to provide a hard disk health assessment and early warning method, device and medium based on multi-model integration for achieving real-time health monitoring of each hard disk node in a service cluster.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] A hard disk health assessment and early warning method based on multi-model integration includes the following steps:

[0011] Obtain the real-time SMART log files of each hard disk node in the service cluster for preprocessing;

[0012] The pre-processed SMART log file is input into the large model for processing, and the health score of each hard disk node is output, wherein the large model is integrated by multiple models;

[0013] Determine whether the health score of each hard disk node exceeds the dynamic health threshold, if so, output a warning signal, if not, do not output a warning signal.

[0014] Furthermore, the pre-processing step comprises:

[0015] Extract the initial key fields:

[0016] Based on the SMART log file, NA values ​​and values ​​that are identical in the entire column are removed, and then correlation tests and normalization processes are performed in sequence to obtain initial key fields;

[0017] Data enhancement and balancing:

[0018] Based on the initial key fields, the SMOTE technology is used to perform data enhancement, and the undersampling or cluster balancing technology is further used for further processing to obtain the final key fields.

[0019] Furthermore, the initial key fields include SMART5, SMART187, SMART188, SMART197, and SMART198.

[0020] Furthermore, the training optimization steps of the large model include:

[0021] Construct training samples: obtain the SMART log files of the hard disk node before normal operation and failure, perform preprocessing, obtain key fields, and use them as training samples;

[0022] Multi-model integration and dynamic feature selection:

[0023] Based on the training samples, multiple classification models in the large model are used to extract their respective dynamic features, and fusion prediction is performed to output the prediction result, i.e., the health score;

[0024] Model training and adaptive optimization:

[0025] An adaptive gradient optimization algorithm is used to optimize and update the weights of the large model, and a stepwise regularization method is used during the training process to eliminate insignificant dynamic features in each round of iteration, while increasing the weights of fault samples in the training samples during the training process.

[0026] Furthermore, the multiple classification models in the large model include a Logistic regression model, a random forest and a support vector machine, respectively, wherein the Logistic regression model serves as a baseline model, and the random forest and the support vector machine serve as auxiliary models.

[0027] Furthermore, the step of outputting the prediction result includes:

[0028] Based on the training samples, the first-layer Logistic regression model, random forest and support vector machine are used to extract respective dynamic features, wherein the random forest selects the dynamic features by calculating the importance scores of key fields when extracting the dynamic features, and uses L1 regularization to select the optimal feature combination, and the support vector machine further extracts features from the optimal feature combination to obtain the dynamic features extracted by the support vector machine;

[0029] The extracted dynamic features are input into the second-layer Logistic regression model, and the Stacking strategy is used for fusion prediction to output the prediction results.

[0030] Furthermore, the dynamic health threshold is set using an adaptive threshold adjustment mechanism, wherein the adaptive threshold adjustment mechanism is based on dynamic adjustment of historical health data of the hard disk node, and sets the dynamic health threshold of the hard disk node as a personalized parameter for dynamic adjustment.

[0031] Furthermore, the large model is also integrated with an incremental learning mechanism, and the incremental learning mechanism is used to periodically update the large model incrementally by manually annotating unlabeled training samples.

[0032] The present invention also provides an electronic device, characterized in that it includes: one or more processors; a memory; and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the hard disk health assessment and early warning method based on multi-model integration as described above.

[0033] The present invention also provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the hard disk health assessment and early warning method based on multi-model integration as described above.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) The present invention collects the SMART log files of each hard disk node in the service cluster in real time, uses a large model after multi-model integration to process, obtains the health score of each hard disk node, and determines whether each hard disk node is healthy in one step. The present invention monitors each hard disk node in the service cluster in real time to ensure real-time control of the health status of each hard disk node.

[0036] (2) The present invention uses a large model after multi-model integration for processing. The large model performs fusion prediction through the features extracted by multiple models, which can not only obtain a more comprehensive feature representation, but also make the large model more adaptable to the characteristics of complex data, thereby improving the generalization ability, accuracy and robustness of the large model prediction.

[0037] (3) Since the traditional fixed threshold may not be able to adapt to the individual differences of different hard disks, the present invention sets a dynamic health threshold by introducing an adaptive threshold adjustment mechanism, thereby further increasing the accuracy of the early warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0039] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0040] Example 1

[0041] This embodiment provides a hard disk health assessment and early warning method based on multi-model integration, such as Figure 1 As shown, the method comprises the following steps:

[0042] Step 1: Collect the SMART log files of each disk node in the service cluster during normal operation and before a failure occurs.

[0043] Step 2: Preprocess the obtained data.

[0044] (1). Extract the initial key fields:

[0045] Including removing NA values, removing the same values ​​in the entire column, correlation test and normalization processing, the initial key fields are obtained: SMART5 (Relocated Sector Count), SMART187 (Reported Uncorrectable Errors), SMART188 (Command Timeout), SMART197 (Current Pending Sector Count), SMART198 (Offline Uncorrectable Sector Count);

[0046] (2) Data enhancement and balancing:

[0047] In order to solve the problem of scarce hard disk failure data and imbalanced categories, the synthetic minority oversampling technology (SMOTE) is introduced in the training process to increase the number of samples. By synthesizing minority pseudo samples, the problem of insufficient failure data is effectively alleviated. In addition, undersampling or cluster balancing technology is used to further enhance the model's ability to classify normal and fault states, and improve the reliability of fault warning.

[0048] Step 3: Use the large model after multi-model integration to train the preprocessed data to improve the accuracy and stability of fault prediction.

[0049] The specific process includes the following:

[0050] (1) Multi-model integration and dynamic feature selection:

[0051] The Logistic regression model is used as the baseline model, and the Random Forest and Support Vector Machine (SVM) are introduced as auxiliary models to achieve multi-model integration. Support Vector Machine (SVM) mainly processes nonlinear features in the data and is particularly suitable for identifying samples with complex feature relationships. Specifically, SVM generates a high-dimensional feature space through nonlinear mappings such as the radial basis kernel function (RBF) to enhance classification capabilities. The SVM model will further discriminate the key fields selected by the random forest to provide a more representative classification.

[0052] During the feature selection process, we first screened key fields using the feature importance score of the random forest, and then used L1 regularization to automate feature selection to find the optimal feature combination. Finally, we used the stacking strategy to fuse multiple model outputs to improve the accuracy and robustness of the prediction results.

[0053] The specific fusion steps are as follows:

[0054] 1) The first layer of base models: It consists of random forest, support vector machine (SVM) and logistic regression models. First, these three models are trained independently on the data to generate initial prediction outputs, i.e., the output features of the first layer. In this way, by combining multiple models to process data from different perspectives, a more comprehensive feature representation can be obtained.

[0055] 2) Second-layer fusion model: The output of the first-layer base model is used as a new input feature and input into the second-layer Logistic regression model. The second-layer Logistic regression model is trained and predicted on this basis, playing a role in comprehensive evaluation. This hierarchical processing can make the model more adaptable to the characteristics of complex data and further improve the stability and generalization ability of prediction.

[0056] 3) Output the final prediction result: The output of the second layer of Logistic regression is the final prediction result of the entire Stacking integrated model. This Stacking method not only integrates the advantages of random forest, SVM and Logistic regression, but also makes the model more accurate and robust in data processing and prediction.

[0057] This fusion approach combines the strengths of multiple models to achieve more comprehensive and accurate prediction and evaluation in hard drive health management.

[0058] (2) Model training and adaptive optimization:

[0059] The weights of the Logistic model are optimized and updated based on the adaptive gradient optimization algorithm (AdaGrad) to adapt to the learning rate of different features. The step-by-step regularization method is used during the training process to remove insignificant features in each round of iteration to improve the model's ability to focus on key fields. In addition, the weights of faulty samples are gradually increased during training to make the model more sensitive to potential hard disk failure trends.

[0060] (3) Threshold adaptive adjustment and dynamic feedback:

[0061] Traditional fixed thresholds may not be able to adapt to the individual differences of different hard disks, so an adaptive threshold adjustment mechanism is introduced in the fault warning system. Through dynamic adjustment based on historical health data, the threshold of hard disk health is set as a personalized parameter to enhance the accuracy of warning. At the same time, a feedback mechanism is added to learn the data of hard disk health status changing over time in real time, and gradually optimize the threshold judgment criteria of the model.

[0062] (4) Active learning and incremental model updating:

[0063] An active learning module is established to guide model updates using unlabeled hard disk data. Through the active learning algorithm, the model selects samples with higher uncertainty and updates the model after the system receives the manually labeled results. This incremental learning mechanism can significantly improve the adaptability of the model and ensure that the system continuously optimizes the accuracy and timeliness of fault warnings over time. At the same time, the periodic incremental update method is adopted to enable the model to have the ability to continuously evolve to adapt to changes in hard disk hardware.

[0064] (5) Edge computing compatibility and adaptive algorithm optimization:

[0065] In view of the deployment requirements of the model on edge devices (such as hard disk management terminals), the algorithm's computational workload and memory usage are further optimized. By quantizing weights, optimizing memory access strategies, and using low-bit floating-point representations (such as float16), the model is adapted to the low power consumption requirements of hardware terminals. In the actual application of edge devices, an adaptive learning rate optimization strategy is adopted to enable the model to achieve more efficient training and prediction under limited computing resources.

[0066] Step 4: Input the hard disk data of each hard disk node collected in real time in the service cluster into the integrated large model to obtain a return value between 0 and 1. The closer the value is to 1, the worse the health of the hard disk. Otherwise, it means that the status is good.

[0067] Step 5: Set the hard disk health status threshold. If the threshold is exceeded, the hard disk will be warned and a warning signal will be output.

[0068] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0069] Example 2

[0070] This embodiment provides an electronic device, characterized in that it includes: one or more processors; a memory; and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the hard disk health assessment and early warning method based on multi-model integration as described in the above-mentioned embodiment 1.

[0071] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0072] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0073] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0075] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0076] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A hard disk health assessment and early warning method based on multi-model integration, characterized in that: The following steps are involved: Obtain the real-time SMART log files of each hard disk node in the service cluster for preprocessing; The pre-processed SMART log file is input into the large model for processing, and the health score of each hard disk node is output, wherein the large model is integrated by multiple models; Determine whether the health score of each hard disk node exceeds the dynamic health threshold, if so, output a warning signal, if not, do not output a warning signal.

2. According to claim 1, a hard disk health assessment and early warning method based on multi-model integration is characterized in that: The pre-processing step comprises: Extract the initial key fields: Based on the SMART log file, NA values ​​and values ​​that are identical in the entire column are removed, and then correlation tests and normalization processes are performed in sequence to obtain initial key fields; Data enhancement and balancing: Based on the initial key fields, the SMOTE technology is used to perform data enhancement, and the undersampling or cluster balancing technology is further used for further processing to obtain the final key fields.

3. The hard disk health assessment and early warning method based on multi-model integration according to claim 2 is characterized in that: The initial key fields include SMART5, SMART187, SMART188, SMART197, and SMART198.

4. The hard disk health assessment and early warning method based on multi-model integration according to claim 1 is characterized in that: The training optimization steps of the large model include: Construct training samples: obtain the SMART log files of the hard disk node before normal operation and failure, perform preprocessing, obtain key fields, and use them as training samples; Multi-model integration and dynamic feature selection: Based on the training samples, multiple classification models in the large model are used to extract their respective dynamic features, and fusion prediction is performed to output the prediction result, i.e., the health score; Model training and adaptive optimization: An adaptive gradient optimization algorithm is used to optimize and update the weights of the large model, and a stepwise regularization method is used during the training process to eliminate insignificant dynamic features in each round of iteration, while increasing the weights of fault samples in the training samples during the training process.

5. The hard disk health assessment and early warning method based on multi-model integration according to claim 4 is characterized in that: The multiple classification models in the large model include Logistic regression model, random forest and support vector machine respectively, wherein the Logistic regression model is used as a baseline model, and the random forest and support vector machine are used as auxiliary models.

6. The hard disk health assessment and early warning method based on multi-model integration according to claim 5 is characterized in that: The step of outputting the prediction result comprises: Based on the training samples, the first-layer Logistic regression model, random forest and support vector machine are used to extract respective dynamic features, wherein the random forest selects the dynamic features by calculating the importance scores of key fields when extracting the dynamic features, and uses L1 regularization to select the optimal feature combination, and the support vector machine further extracts features from the optimal feature combination to obtain the dynamic features extracted by the support vector machine; The extracted dynamic features are input into the second-layer Logistic regression model, and the Stacking strategy is used for fusion prediction to output the prediction results.

7. The hard disk health assessment and early warning method based on multi-model integration according to claim 1 is characterized in that: The dynamic health threshold is set using an adaptive threshold adjustment mechanism, wherein the adaptive threshold adjustment mechanism is based on dynamic adjustment of historical health data of the hard disk node, and sets the dynamic health threshold of the hard disk node to a personalized parameter for dynamic adjustment.

8. The hard disk health assessment and early warning method based on multi-model integration according to claim 1 is characterized in that: The large model is also integrated with an incremental learning mechanism, which is used to periodically and incrementally update the large model by manually annotating unlabeled training samples.

9. An electronic device, characterized in that: include: one or more processors; Memory; and One or more programs stored in a memory, wherein the one or more programs include instructions for executing the hard disk health assessment and early warning method based on multi-model integration as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: It includes one or more programs executed by one or more processors of an electronic device, and the one or more programs include instructions for executing the hard disk health assessment and early warning method based on multi-model integration as described in any one of claims 1-8.

Images