A hard disk health assessment method and storage device

Abstract

The embodiment of the application provides a hard disk health assessment method and device, the method comprises the following steps: acquiring data of a plurality of indexes related to health degree of a hard disk at a specified use time; inputting the data into a plurality of different models; and determining the health degree of the hard disk at the specified use time according to the output of the plurality of models. The hard disk health degree assessment method provided by the embodiment of the application fuses the output of a plurality of models to obtain stable and accurate health degree indexes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage, in particular to a hard disk health assessment method and a storage device. BACKGROUND

[0002] With the rapid development of big data, cloud computing and artificial intelligence technology, the demand for high-reliability storage systems is increasing. The reliability of hard disks in the storage system has become one of the important factors restricting the reliability of the storage system. Hard disk failure can cause user data loss and damage, system read-write performance degradation, storage system paralysis, etc. Therefore, how to accurately predict the health and life of the hard disk has become a growing concern.

[0003] In the hard disk, the SMART data is usually recorded by monitoring and recording a plurality of parameters in the hard disk in real time according to the Self-Monitoring, Analysis and Reporting Technology (SMART), so that each hard disk can be monitored based on the SMART data.

[0004] In related technologies, the health of the hard disk is usually evaluated by the Euclidean distance method or the linear evaluation method. In the Euclidean distance method, the health of the hard disk is measured based on the distance between the SMART data of the hard disk and the threshold data. In the linear evaluation method, the health of the hard disk is predicted according to the linear function of the health of the hard disk and time. However, the health of the hard disk obtained by the above methods cannot stably indicate the actual health of the hard disk and has a large error. SUMMARY

[0005] The embodiments of the present application aim to provide a hard disk health evaluation scheme, which fuses the output values of a plurality of anomaly detection models based on the SMART data of the hard disk to obtain a stable and accurate health index.

[0006] To achieve the above-mentioned purpose, the first aspect of the present application provides a hard disk health assessment method, which is executed by a storage device and includes: acquiring data of a plurality of indexes related to health degree of a hard disk at a specified use time; inputting the data into a plurality of different models; and determining the health degree of the hard disk at the specified use time according to the outputs of the plurality of models.

[0007] By determining the health of the hard disk according to the outputs of a plurality of different models, the characteristics of each model are comprehensively considered, and a stable and accurate health index is provided.

[0008] In a possible implementation manner of the first aspect, the determining the health degree of the hard disk at the specified use time according to the outputs of the plurality of models specifically comprises determining the health degree of the hard disk at the specified use time based on a weighted sum of the outputs of the plurality of models.

[0009] The outputs of the plurality of models are fused by performing a weighted sum on the outputs of the plurality of models, and the weights of the outputs of the plurality of models can be equal, or can be unequal, or can be dynamically adjusted according to different scenarios.

[0010] In a possible implementation manner of the first aspect, each model in the plurality of models is trained based on an anomaly detection algorithm, and the anomaly detection algorithms used by the plurality of models are different.

[0011] The plurality of models are trained based on different anomaly detection algorithms, and since the anomaly detection algorithm is unsupervised learning, manual labeling of samples is not required, thereby saving the labor cost, and meanwhile, the anomaly detection model can provide high prediction accuracy.

[0012] In a possible implementation manner of the first aspect, the number of the plurality of models is three, and the anomaly detection algorithms used by the three models are an isolation forest algorithm, a local outlier factor algorithm and a K-means clustering algorithm respectively.

[0013] In a possible implementation manner of the first aspect, the plurality of models are sent to the storage device by a training device or are trained by the storage device, and the training device is configured to train the plurality of models.

[0014] In a possible implementation manner of the first aspect, the plurality of models are trained by sampling data of a plurality of indexes related to the health degree of the hard disk in a preset use time period before the end of the life of the hard disk.

[0015] The anomaly detection model is trained by using the sampling data of the plurality of indexes of the hard disk in a period before the end of the life of the hard disk, since the storage device usually samples the SMART data of the hard disk in the second half of the use life in practice, the sampling data is easier to obtain, and in addition, since the sampling data corresponds to a lower health degree of the hard disk, the anomaly degree output by the anomaly detection model can be positively correlated with the health degree, for example, the anomaly degree can be directly used as the health degree, thereby reducing the calculation cost of the health degree.

[0016] In a possible implementation manner of the first aspect, the hard disk comprises a target disk, and the method further comprises: obtaining or generating a first data set of a plurality of reference disks, the first data set comprising health degrees of the plurality of reference disks at a plurality of use times; generating a second data set of the target disk, the second data set comprising health degrees of the target disk at a plurality of use times, a time span of the plurality of use times in the first data set being greater than a time span of the plurality of use times in the second data set; selecting a reference disk according to a similarity of the health degrees of the plurality of use times corresponding between the first data set and the second data set; and predicting a health degree of the target disk at a specified future time according to the first data set of the selected reference disk.

[0017] By predicting the health degree of the target disk according to the data of the reference disk, the calculation cost can be reduced, and the health degree of the target disk at a future time can be accurately predicted.

[0018] In a possible implementation manner of the first aspect, the predicting the health degree of the target disk at the specified future time according to the first data set of the selected reference disk comprises: fitting a mapping relationship between the health degrees of the first use times of the selected reference disk and the health degrees of the second use times of the target disk, the first use times and the second use times being corresponding times; and predicting the health degrees of the target disk at a plurality of specified future times according to the mapping relationship and the first data set.

[0019] By fitting the mapping relationship between the health degrees of the reference disk and the health degrees of the corresponding use times of the target disk, the accuracy of the predicted health degree of the target disk at the future time can be further improved.

[0020] In a possible implementation manner of the first aspect, the method further comprises: determining, according to the predicted health degrees of the target disk at the plurality of specified future times, a time at which the health degree of the target disk reaches a threshold value, and taking the time at which the health degree reaches the threshold value as an end-of-life time of the target disk.

[0021] By predicting the life span of the target disk according to the predicted health degree of the target disk at the future time, the life span of the target disk can be accurately predicted, so that data backup and the like can be performed in advance to prevent various problems caused by failure of the target disk.

[0022] In a possible implementation manner of the first aspect, the hard disk is a solid state disk.

[0023] The second aspect of the application provides a storage device, comprising: an acquisition unit configured to acquire data of a plurality of indexes related to health degree of a hard disk at a specified use time; an input unit configured to input the data into a plurality of different models; and a determination unit configured to determine the health degree of the hard disk at the specified use time according to outputs of the plurality of models.

[0024] In a possible implementation of the second aspect, the determination unit is specifically configured to determine the health degree of the hard disk at the specified use time based on a weighted sum of the outputs of the plurality of models.

[0025] In a possible implementation of the second aspect, each of the plurality of models is trained based on an anomaly detection algorithm, and the anomaly detection algorithms used by the plurality of models are different.

[0026] In a possible implementation of the second aspect, the number of the plurality of models is three, and the anomaly detection algorithms used by the three models are isolated forest algorithm, local outlier factor algorithm, and K-means clustering algorithm respectively.

[0027] In a possible implementation of the second aspect, the plurality of models are sent to the storage device by a training device or are trained by the storage device, and the training device is configured to train the plurality of models.

[0028] In a possible implementation of the second aspect, the plurality of models are trained by sampling data of a plurality of indexes related to health degree of a failed hard disk in a preset use period before the end of the life of the failed hard disk.

[0029] In a possible implementation of the second aspect, the hard disk comprises a target disk, and the storage device further comprises: an acquisition or generation unit configured to acquire or generate a first data set of a plurality of benchmark disks, the first data set comprising health degrees of the benchmark disks at a plurality of use times; a generation unit configured to generate a second data set of the target disk, the second data set comprising health degrees of the target disk at a plurality of use times, a time span of the plurality of use times in the first data set being greater than a time span of the plurality of use times in the second data set; a selection unit configured to select a benchmark disk according to a similarity of health degrees of the plurality of use times corresponding to the first data set and the second data set; and a prediction unit configured to predict a health degree of the target disk at a specified time in the future according to the first data set of the selected benchmark disk.

[0030] In a possible implementation manner of the second aspect, the prediction unit is specifically configured to: fit a mapping relationship between the health degree of the selected first use time of the target disk and the health degree of the second use time of the target disk, the first use time and the second use time being corresponding times; and predict the health degrees of the target disk at the future multiple specified times according to the mapping relationship and the first data set.

[0031] In a possible implementation manner of the second aspect, the determination unit is further configured to determine, according to the predicted health degrees of the target disk at the future multiple specified times, a time at which the health degree of the target disk reaches a threshold value, and take the time at which the health degree reaches the threshold value as the end-of-life time of the target disk.

[0032] The third aspect of the present application provides a storage device, characterized by comprising a processor and a memory, the memory storing executable computer program instructions, and the processor executing the executable computer program instructions to implement the method of the first aspect or the possible implementation manners of the first aspect.

[0033] The fourth aspect of the present application provides a computer readable storage medium, characterized by storing computer program instructions, when the computer program instructions are executed in a computer or a processor, causing the computer or the processor to execute the method of the first aspect or the possible implementation manners of the first aspect.

[0034] The fifth aspect of the present application provides a computer program product, comprising computer program instructions, when the computer program instructions are executed in a computer or a processor, causing the computer or the processor to execute the method of the first aspect or the possible implementation manners of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0035] The embodiments of the present application can be made more clear by describing the embodiments of the present application in combination with the accompanying drawings:

[0036] Figure 1A An architecture diagram of a centralized storage system 120 of a disk control separation structure applied to the embodiments of the present application;

[0037] Figure 1B An architecture diagram of a centralized storage system 120 of a disk control integration structure applied to the embodiments of the present application;

[0038] Figure 1C An architecture diagram of a distributed storage system applied to the embodiments of the present application;

[0039] Figure 2 A system architecture schematic diagram provided by the embodiments of the present application for training a hard disk anomaly detection model;

[0040] Figure 3 A method flowchart for training an abnormality detection model provided by an embodiment of the present application;

[0041] Figure 4 A method flowchart for hard disk health assessment provided by an embodiment of the present application;

[0042] Figure 5 A schematic diagram of health degree curves of various hard disks provided by an embodiment of the present application

[0043] Figure 6 A schematic diagram of health degree curves of various hard disks provided by an embodiment of the present application Figure 5 An enlarged view of one health degree curve in the schematic diagram of health degree curves of various hard disks provided by an embodiment of the present application

[0044] Figure 7 A method flowchart for predicting hard disk health degree provided by an embodiment of the present application;

[0045] Figure 8 A schematic diagram of predicted hard disk health degree curves provided by an embodiment of the present application

[0046] Figure 9 An architecture diagram of a storage device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0048] The hard disk health assessment scheme provided by the embodiments of the present application can be applied to a storage system. The storage system includes a centralized storage system and a distributed storage system. The centralized storage system refers to a center node composed of one or more host devices, and data is stored in the center node in a centralized manner, and all data processing services of the entire system are deployed on the center node. The distributed storage system refers to a system in which data is stored in multiple independent storage nodes. Users can access data in the storage nodes through application programs. The computer running these application programs is called an "application server". The application server can be a physical machine or a virtual machine. The physical application server includes but is not limited to desktop computers, servers, notebook computers, and mobile devices. The application server can access the storage nodes through a fiber switch to access data. Among them, the switch is only an optional device, and the application server can also communicate with the storage nodes directly through the network.

[0049] Figure 1A An architecture diagram of a centralized storage system 120 with disk control separation structure applied by an embodiment of the present application. The storage system 120 is connected with a plurality of hosts 200, for example, application servers, which are all connected with the storage system 120 to access data in the storage system 120. Figure 1AThe centralized storage system shown is characterized by having a unified entry point through which all data from host 200 passes. This entry point is, for example, engine 121 in storage system 120.

[0050] like Figure 1A As shown, engine 121 contains one or more controllers. Figure 1A Let's take an engine with two controllers as an example. Controller 0 and Controller 1 have a mirror channel. When Controller 0 writes data to its memory 124, it can send a copy of that data to Controller 1 through the mirror channel. Controller 1 then stores the copy in its local memory 124. Thus, Controller 0 and Controller 1 act as backups for each other. When Controller 0 fails, Controller 1 can take over its operations, and vice versa, preventing hardware failures from rendering the entire storage system 120 unavailable. When four controllers are deployed in engine 121, any two controllers have a mirror channel, thus any two controllers act as backups for each other.

[0051] Engine 121 also includes a front-end interface 125 and a back-end interface 126. The front-end interface 125 is used to communicate with the application server to provide storage services to the application server. The back-end interface 126 is used to communicate with the hard disk 134 to expand the capacity of the storage system. Through the back-end interface 126, engine 121 can connect to more hard disks 134, thus forming a very large storage resource pool.

[0052] In terms of hardware, such as Figure 1A As shown, controller 0 includes at least processor 123 and memory 124. Processor 123 is a central processing unit (CPU) used to process data access requests from outside the storage system (server or other storage systems), and also to process requests generated within the storage system. For example, when processor 123 receives write data requests from the server through the front-end port, it temporarily stores the data in these write data requests in memory 124. When the total amount of data in memory 124 reaches a certain threshold, processor 123 sends the data stored in memory 124 to hard disk 134 for persistent storage through back-end port 126.

[0053] The memory 124 refers to an internal memory that exchanges data directly with the processor 123, which can read and write data at any time and at a very fast speed, as temporary data storage for the operating system or other programs running. The memory 124 includes at least two types of memories, for example, the memory can be a random access memory or a read-only memory (Read Only Memory, ROM). For example, the random access memory is a dynamic random access memory (Dynamic Random Access Memory, DRAM) or a storage class memory (Storage Class Memory, SCM). The DRAM is a semiconductor memory, which is a kind of volatile memory device, like most random access memories (Random Access Memory, RAM). The SCM is a composite storage technology that combines the characteristics of traditional storage devices and memories. The storage class memory can provide faster read and write speeds than hard disks, but slower computing speeds than DRAM, and is more cost-effective than DRAM. However, the DRAM and SCM are only exemplary in this embodiment, and the memory can also include other random access memories, such as static random access memories (Static Random Access Memory, SRAM), etc. For read-only memories, for example, they can be programmable read-only memories (Programmable Read Only Memory, PROM), erasable programmable read-only memories (Erasable Programmable Read Only Memory, EPROM), etc. In addition, the memory 124 can also be a dual in-line memory module or a dual-line memory module (Dual In-line Memory Module, DIMM), i.e., a module composed of dynamic random access memory (DRAM), and can also be a solid state disk (Solid State Disk, SSD). In practical applications, multiple memories 124 and different types of memories 124 can be configured in the controller 0. This embodiment does not limit the number and type of the memory 124. In addition, the memory 124 can be configured to have a power retention function. The power retention function refers to the data stored in the memory 124 will not be lost when the system is powered off and then powered on again. The memory with the power retention function is called a non-volatile memory.

[0054] The memory 124 stores software programs, and the processor 123 runs these software programs to manage the hard drive. This hard drive management includes, for example, abstracting the hard drive into a storage resource pool and then dividing it into LUNs for server use. Here, the LUN is essentially the hard drive seen on the server. Of course, some centralized storage systems are also file servers themselves, providing shared file services to the server.

[0055] Controller 1 (and others) Figure 1A The hardware components and software structure of the controller (not shown) are similar to those of controller 0, and will not be described in detail here.

[0056] exist Figure 1A In the disk-controller-separated storage system shown, engine 121 may not have a hard drive bay; hard drive 134 needs to be placed in hard drive array 130, and back-end interface 126 communicates with hard drive array 130. Back-end interface 126 exists in the form of an adapter card within engine 121, and two or more back-end interfaces 126 can be used simultaneously on one engine 121 to connect multiple hard drive arrays. Alternatively, the adapter card can be integrated onto the motherboard, in which case it can communicate with processor 123 via the PCIe bus.

[0057] It should be noted that, Figure 1A Only one engine 121 is shown in the figure. However, in actual applications, the storage system may contain two or more engines 121, and redundancy or load balancing may be performed between multiple engines 121.

[0058] The hard disk array 130 includes a control unit 131 and a plurality of hard disks 134. The control unit 131 can have various forms. In one case, the hard disk array 130 is a smart drive enclosure, such as... Figure 1AAs shown, the control unit 131 includes a CPU and a memory. The CPU is used to perform address translation and read / write data operations. The memory is used to temporarily store data to be written to the hard disks 134 or data read from the hard disks 134 to be sent to the controller. Alternatively, the control unit 131 is a programmable electronic component, such as a data processing unit (DPU). The DPU has the versatility and programmability of the CPU, but is more specialized and can efficiently operate on network packets, storage requests, or analysis requests. The DPU is distinguished from the CPU by a greater degree of parallelism (to handle a large number of requests). Alternatively, the DPU here can also be replaced by a graphics processing unit (GPU), an embedded neural-network processing unit (NPU), or the like. Generally, the number of control units 131 can be one, two, or more. When the hard disk array 130 contains at least two control units 131, the hard disks 134 have an ownership relationship with the control units 131, and each control unit can only access the hard disks belonging to it. Therefore, this often involves forwarding read / write data requests between the control units 131, resulting in a longer data access path. In addition, if the storage space is insufficient, when a new hard disk 134 is added to the hard disk array 130, the ownership relationship between the hard disk 134 and the control unit 131 needs to be re-bound, which is complicated and results in poor scalability of the storage space. Therefore, in another embodiment, the functions of the control unit 131 can be offloaded to the network card 104. In other words, in this embodiment, the hard disk array 130 does not have a control unit 131 inside, but the data read / write, address translation, and other computing functions are completed by the network card 104. At this time, the network card 104 is a smart network card. It can contain a CPU and a memory. The CPU is used to perform address translation and read / write data operations. The memory is used to temporarily store data to be written to the hard disks 134 or data read from the hard disks 134 to be sent to the controller. It can also be a programmable electronic component, such as a data processing unit (DPU). The DPU has the versatility and programmability of the CPU, but is more specialized and can efficiently operate on network packets, storage requests, or analysis requests. The DPU is distinguished from the CPU by a greater degree of parallelism (to handle a large number of requests). Alternatively, the DPU here can also be replaced by a graphics processing unit (GPU), an embedded neural-network processing unit (NPU), or the like.There is no belonging relationship between the network card 104 and the hard disk 134 in the hard disk array 130, and the network card 104 can access any hard disk 134 in the hard disk array 130, so that the hard disk is convenient to expand when the storage space is insufficient.

[0059] The hard disk 134 can be an SSD, or can be a mechanical hard disk (i.e., a magnetic disk). Compared with a traditional magnetic disk, the SSD has the characteristics of fast startup, fast read and write, fixed read time, wide working temperature range, and no noise, and the SSD will not have mechanical failure caused by mechanical component activity, and has high safety and reliability. The hard disk health evaluation method provided in the embodiments of the present application is suitable for health evaluation of an SSD. It can be understood that the hard disk health evaluation method provided in the embodiments of the present application is also suitable for health evaluation of a magnetic disk.

[0060] According to the type of the communication protocol between the engine 121 and the hard disk array 130, the hard disk array 130 can be a SAS hard disk array, can be an NVMe hard disk array, and can be other types of hard disk arrays. The SAS hard disk array adopts the SAS3.0 protocol, and each frame supports 25 SAS hard disks. The engine 121 is connected with the hard disk array 130 through a built-in SAS interface or a SAS interface module. The NVMe hard disk array is more like a complete computer system, and the NVMe hard disk is inserted into the NVMe hard disk array. The NVMe hard disk array is connected with the engine 121 through an RDMA port.

[0061] It can be understood that, Figure 1A Although the centralized storage system with the disk control separation structure is shown in the foregoing embodiment, the application range of the embodiments of the present application is not limited thereto, for example, the embodiments of the present application can also be applied to the centralized storage system with the disk control integration structure shown in FIG. 2. Figure 1B The centralized storage system with the disk control integration structure is different from the disk control separation structure in that the engine 121 has a hard disk slot, the hard disk 134 can be directly deployed in the engine 121, and the back-end interface 126 is an optional configuration. When the storage space of the system is insufficient, more hard disks or hard disk arrays can be connected through the back-end interface 126.

[0062] The embodiments of the present application can also be applied to the distributed storage system shown in FIG. 3. Figure 1C The distributed storage system includes a storage node cluster. The storage node cluster includes one or more storage nodes 20( Figure 1CThree storage nodes 20a, 20b and 20c are shown in FIG. 1, but the number of storage nodes is not limited to three. Each storage node 20 can be interconnected with other storage nodes 20. Each storage node 20 is connected to a plurality of hosts 200. Each host 200 is connected to and interacts with a plurality of storage nodes 20 to store data in the plurality of storage nodes 20, thereby achieving reliable storage of data. Each storage node 20 includes at least a processor 201, a memory 202 and a hard disk 203. The processor 201, the memory 202 and the hard disk 203 have the same structure and function as the processor 123, the memory 124 and the hard disk 134 in FIG. 1, respectively. For details, refer to the related description in FIG. 1, which will not be repeated here. Figure 1A The processor 123, the memory 124 and the hard disk 134 in FIG. 1 have the same structure and function as the processor 201, the memory 202 and the hard disk 203 in FIG. 1, respectively. For details, refer to the related description in FIG. 1, which will not be repeated here. Figure 1A

[0063] In a storage system, SMART data of each hard disk can be periodically sampled, so that the health of the hard disk can be monitored according to the SMART data, and the failure of the hard disk can be predicted. The SMART data includes, for example, values of a plurality of indicators such as power-on time, on-off count, uncorrectable error number, newly added bad sector number, newly added bad block number, total erase count, etc. Each SMART indicator is usually provided with a threshold value. If a certain SMART data of a hard disk approaches the threshold value, it indicates that the hard disk will become unreliable, for example, may cause data loss or hard disk failure.

[0064] In the related art, methods for evaluating the health of a hard disk include binary method, Euclidean distance method and linear evaluation method. In the binary method, the health of a hard disk includes two states of health and failure, i.e., the method can predict whether the hard disk will fail and when the hard disk will fail, but the method cannot quantitatively predict the real-time health of the hard disk. In the Euclidean distance method, the health of a hard disk is determined according to the distance between the SMART data of the hard disk and the threshold value of the corresponding SMART indicator. However, due to the characteristics of the actual running data of the hard disk, a single distance-based method is difficult to obtain a stable and accurate health evaluation indicator. In the linear evaluation method, a linear correlation function of the health of a hard disk and time is established. However, the actual health of a hard disk is simultaneously affected by factors such as read-write frequency and environmental temperature, and has no strict strong correlation with the time factor, so the health obtained by the method has a large error.

[0065] The embodiments of the present application provide a method for effectively evaluating the health of a hard disk. In a storage system, a plurality of anomaly detection models are used to make predictions based on the SMART data of the hard disk, and the outputs of the plurality of anomaly detection models are fused to obtain a more accurate health of the hard disk, wherein the plurality of anomaly detection models are obtained by training based on the SMART data of the hard disk. The method can be used in the storage system 120 shown in FIG. 2, Figure 1A the storage system 120 shown in FIG. 3, or Figure 1B the storage system 120 shown in FIG. 4.​Figure 1C The storage system shown in the text executes, and will be referred to below as... Figure 1A The description will be based on the storage system 120.

[0066] Figure 2 This is a schematic diagram of a system architecture for training an anomaly detection model provided in an embodiment of this application. The anomaly detection model has model parameters corresponding to the anomaly detection algorithm. These parameters are determined by training the model using multiple training samples based on the anomaly detection algorithm. The trained anomaly detection model can output the anomaly score of a sample based on its features. This anomaly score is the degree of difference between the sample being tested and most samples in the training sample set. Therefore, in the case where the anomaly detection model is trained using multiple hard drive samples corresponding to high health scores, the anomaly score of the hard drive sample output by the model is negatively correlated with the hard drive health score. Alternatively, in the case where the anomaly detection model is trained using multiple hard drive samples corresponding to low health scores, the anomaly score of the hard drive sample output by the model is positively correlated with the hard drive health score. Thus, the trained anomaly detection model can be used to predict the health score of hard drives.

[0067] like Figure 2 As shown, the system architecture includes: a training device 210, used to acquire SMART data from the hard drive of the storage system 120, and use the SMART data to train multiple different anomaly detection models. The training process of the multiple anomaly detection models will be described in detail below. Figure 3 The following description is provided. After training multiple anomaly detection models, the training device 210 sends these models to the storage system 120, allowing the storage system 120 to use them to predict hard drive health. It is understood that the training device 210 can connect to multiple storage systems to obtain SMART data from the hard drives in each system for model training. In another embodiment, the storage system 120 can transmit the hard drive SMART data to a database on the storage device, and the training device 210 can read the hard drive data from the database for model training. In another embodiment, the storage system 120 can train the model using its own CPU 123 to obtain an anomaly detection model. In yet another embodiment, the storage system 120 can train the model using a computing chip (such as a Field Programmable Gate Array (FPGA) chip) plugged into the storage system 120 to obtain an anomaly detection model. In the following description, the model will be trained using... Figure 2 The system architecture shown is used as an example for description.

[0068] exist Figure 2 In the storage system 120 shown, CPU 123 and memory 124 can beFigure 1A CPU and memory in controller 0, and also for CPU and memory in controller 1, hereinafter taking controller 0 as an example for description.

[0069] The storage system 120 can collect SMART data of each hard disk 134 included therein periodically during operation, and can send specific SMART data to the training device 210 according to the need of model training, which will be described in detail below with reference to Figure 3 DETAILED DESCRIPTION.

[0070] After receiving the plurality of anomaly detection models from the training device 210, the storage system 120 can store the plurality of anomaly detection models in a persistent storage, such as the hard disk 134. When the plurality of anomaly detection models 220 are needed to be used for predicting the health of the hard disk, the storage system 120 can read the plurality of anomaly detection models from the hard disk 134 and store the plurality of anomaly detection models in the memory 124 for the CPU 123 to read and run the plurality of models. The CPU 123 can periodically obtain the SMART data of each hard disk 134 at a specified use time (for example, the current time) and input the SMART data into the plurality of anomaly detection models respectively, and then the CPU 123 fuses the outputs of the plurality of anomaly detection models, so as to obtain the health of each hard disk 134 at the specified use time. Alternatively, when the storage system 120 adds a new hard disk 134, the CPU 123 can obtain the SMART data of the new hard disk 134 at the current time multiple times and input the SMART data into the plurality of anomaly detection models respectively, so that the CPU 123 can predict the health of the newly added hard disk 134 at the specified use time based on the outputs of the plurality of anomaly detection models, which will be described in detail below with reference to Figure 4 DETAILED DESCRIPTION.

[0071] Figure 3 The method for training an anomaly detection model provided by the embodiments of the present application is shown in the flowchart, which can be executed by the training device 210 in the storage system 120, and includes the following steps: Figure 2

[0072] Step S301, receiving the SMART data of the hard disk at at least one use time from the storage system;

[0073] Step S302, training an anomaly detection model based on the SMART data.

[0074] Each step in the method shown in the following will be described in detail. Figure 3

[0075] Firstly, in step S301, the SMART data of the hard disk at at least one use time is received from the storage system 120.

[0076] ​​The storage system 120 can send the SMART data of the hard disk to the training device 210 according to the need of model training. The entire SMART data of the hard disk includes the values of a plurality of indexes, for example, more than one hundred indexes. The storage system 120 can select the data of the indexes related to the health and life degradation of the hard disk from the entire SMART data. The selected indexes include, for example, the power-on time, the switch count, the number of uncorrectable errors, the number of newly added bad sectors, the number of newly added bad blocks, the total number of erasures, and the like.

[0077] The storage system 120 can also send the SMART data of a plurality of usage times in a predetermined usage period of the hard disk to the training device 210 according to the training strategy of the model.

[0078] Specifically, in an embodiment, the storage system 120 can select the SMART data of a plurality of hard disks in a predetermined period after the hard disks are enabled to train the anomaly detection model. Since the health of the hard disk is usually the highest (for example, the health at this time can be represented as 1) when the hard disk is newly enabled, the anomaly detection model is trained by using the SMART data in this period, that is, the SMART data corresponding to the high health is taken as the majority of normal samples. The anomaly degree output by the trained anomaly detection model is negatively correlated with the health of the hard disk, that is, the greater the anomaly degree, the worse the health state of the hard disk, that is, the smaller the health.

[0079] In another embodiment, the storage system 120 can select the SMART data of a hard disk in a predetermined usage period before the hard disk fails to train the anomaly detection model after the hard disk fails. Since the health of the hard disk is the lowest (for example, the health at this time can be represented as 0) when the hard disk fails, the anomaly detection model is trained by using the SMART data in this period, that is, the SMART data corresponding to the low health is taken as the majority of normal samples. The anomaly degree output by the trained anomaly detection model is positively correlated with the health of the hard disk, that is, the greater the anomaly degree, the better the health state of the hard disk, that is, the higher the health. In practice, since the hard disk is usually in good health when it is newly enabled, the storage system 120 usually does not collect the SMART data in a predetermined period after the hard disk is enabled, but starts to collect the SMART data after the hard disk has been used for a long time (for example, at the middle time of the life of the hard disk) to monitor the health state of the hard disk. Therefore, the SMART data of the hard disk stored in the storage system 120 usually lacks the data when the hard disk is newly enabled, and has more data before the hard disk fails. The method of training the anomaly detection model in this embodiment is more suitable for this practical situation. In addition, since the anomaly degree output by the anomaly detection model is positively correlated with the health in this embodiment, for example, the anomaly degree can be directly regarded as the health, thereby reducing the calculation cost of the health. Hereinafter, this embodiment will be described in detail.

[0080] The storage system 120 can further pre-process the selected SMART data and send the pre-processed SMART data to the training device 210. For example, the sampling time of the SMART data can be processed to be uniformly distributed. For example, in order to obtain a set of SMART data per day for a hard disk, the SMART data sampled at multiple sampling time points in a day can be averaged to obtain the SMART data corresponding to the day. In the case where the sampling sample of a day is missing, the SMART data of the day can be completed by an interpolation method.

[0081] In step S302, the abnormality detection model is trained based on the SMART data.

[0082] In the embodiments of the present application, a plurality of abnormality detection algorithms based on different principles are adopted, and the plurality of abnormality detection models are trained by the SMART data obtained as described above.

[0083] In an embodiment, the iforest abnormality detection model is trained by an isolation forest (iforest) algorithm. The isolation forest algorithm is an unsupervised abnormality detection algorithm, i.e., no labeled sample is needed for training. Each training sample is, for example, a set of SMART data X = {x1, x2, …, xn} collected at a certain time for a hard disk, which includes the data of a plurality of health degree related indexes (i.e., indexes 1 to n). As described above, the training sample can be a set of SMART data of a hard disk at a certain time in a predetermined period before the failure of the hard disk, and the predetermined period can be a period of time less than a predetermined time length from the time when the hard disk fails. n} collected at a certain time for a hard disk, which includes the data of a plurality of health degree related indexes (i.e., indexes 1 to n). As described above, the training sample can be a set of SMART data of a hard disk at a certain time in a predetermined period before the failure of the hard disk, and the predetermined period can be a period of time less than a predetermined time length from the time when the hard disk fails.

[0084] The training device 210 can use the N samples X to train a plurality of isolated trees, thereby obtaining an iforest anomaly detection model. In training one isolated tree, a part (e.g., Ψ) of the N samples can be randomly selected, the Ψ samples are put into the root node of the isolated tree, a SMART indicator q corresponding to the root node is randomly assigned, and a split value p corresponding to the indicator is randomly determined, the split value p being a value between the maximum value and the minimum value of the indicator q in the current Ψ samples of the root node; the Ψ samples are divided into two child nodes of the root node based on the indicator q and the split value p, for example, samples with a value of the indicator q less than p are divided into the left child node, and samples with a value of the indicator q greater than p are divided into the right child node. Then, the samples in the newly generated nodes can be segmented by a similar process as above, and new child nodes are generated until the last generated child node has only one sample (cannot be further segmented, and the child node is a leaf node) or the isolated tree has grown to a set height, thereby stopping the growth of the isolated tree, and the height is the number of connection edges between the nodes from the leaf node to the root node.

[0085] After obtaining t isolated trees (t is a predetermined number, for example, 100) as described above, the training of the iforest anomaly detection model is completed. Thus, based on the iforest anomaly detection model obtained by the training, the abnormality degree of a to-be-tested sample can be predicted by the following formula (1):

[0086]

[0087] Wherein, x represents a to-be-tested sample, which, like X, includes SMART data of a plurality of indicators related to health, ψ is the number of samples used to train each isolated tree, h(x) is the height of the sample x in each isolated tree, E(h(x)) is the expected value of the height of the sample x in the t isolated trees, and c(ψ) is the average value of the height of the isolated tree given the training sample number ψ, which is used to normalize the expected value of the height of the sample x E(h(x)). As can be seen from formula (1), if the expected value of the height of the to-be-tested sample in the isolated forest model is smaller, the abnormality degree is higher. This is because the smaller the expected value of the height of the to-be-tested sample, the more the to-be-tested sample falls in the region with sparse distribution of training samples, and thus the higher the abnormality degree of the to-be-tested sample compared with the training samples.

[0088] That is, in the case of training an anomaly detection model using a plurality of samples a predetermined period of time before a hard disk failure, the output of the anomaly detection model indicates the abnormality degree of the to-be-tested sample compared with the plurality of samples with lower health, and thus the higher the abnormality degree, the higher the health of the to-be-tested sample.

[0089] In another embodiment, the LOF anomaly detection model is trained by a Local Outlier Factor (LOF) algorithm. The LOF algorithm is a point density-based algorithm. Similarly as the iForest algorithm described above, to train the LOF model, the training device 210 can obtain a plurality of samples X = {x1, x2,..., xn} and distribute the plurality of samples into a space including dimensions 1 to n, so that for a sample x to be tested, the LOF score of the sample x can be calculated by the following equation (2):

[0090]

[0091] wherein the sample x has a corresponding point x in the space, N k (x) represents all points having a distance from the point x within the kth distance, which can be referred to as the kth neighborhood of the point x, p k (p) is a point belonging to the kth neighborhood of the point p in the range of N k (x), p k (x) is the density of points in the kth neighborhood of the point x. According to the equation (2), if the LOF score of the sample x is greater than 1, it indicates that the density of the point x is less than the density of points in its neighborhood, and the sample x can be an abnormal sample, i.e., the abnormality is high, and if the LOF score of the sample x is less than or equal to 1, it indicates that the density of the point x is greater than or equal to the density of points in its neighborhood, and the abnormality of the sample x is small.

[0092] In yet another embodiment, the K-means anomaly detection model is trained by an anomaly detection algorithm based on a K-means clustering algorithm. Similarly, the training device 210 can obtain N samples X = {x1, x2,..., xn} from the database 220, cluster the N samples by the clustering algorithm to obtain the centroid points (i.e., centroid samples) of the classes, and thus obtain the anomaly detection model. When the anomaly detection model is used to predict a sample x to be tested, the distance between the sample x to be tested and the centroid points can be calculated, and the abnormality of the sample x can be determined based on the distance, specifically, the greater the distance, the farther the sample x is from the centroid, and thus the higher the abnormality. In another embodiment, considering that the sizes of the classes are inconsistent, in order to calculate a consistent similarity for the sample to be tested in each class, the abnormality can be calculated by the following equation (3):

[0093]

[0094] wherein p i is the sample to be tested, p j is the N training samples, Dis(p i ) represents the distance of the sample p i from the centroid, representing N training samples p j the mean of the distances to the centroid, var(Dis(p j )) for N training samples p j the variance of the distances to the centroid.

[0095] It can be understood that the anomaly detection model trained in the embodiments of the present application is not limited to the three anomaly detection models described above, but can be any other type of anomaly detection model.

[0096] After the training device 210 obtains the plurality of anomaly detection models by the method shown in Figure 3 , the training device 210 sends the plurality of anomaly detection models to the storage system 120, so that the storage system 120 can predict the health degree of the hard disk by the plurality of anomaly detection models. Specifically, the training device 210 can send the model structure and the model parameters included in each anomaly detection model to the storage system 120. For example, for the iforest anomaly detection model described above, the training device 210 can send the model data of the node structure of each isolated tree, the SMART indicators and the split values corresponding to each non-leaf node, the number of training samples ψ, etc. included in the model to the storage system 120, so that the storage system 120 can use the model data to use the iforest anomaly detection model.

[0097] Figure 4 A flowchart of a hard disk health assessment method provided by the embodiments of the present application, which can be executed by the storage system 120 in Figure 2 , the method comprises:

[0098] Step S401, obtaining data of a plurality of indicators related to health degree of the hard disk at a specified use time;

[0099] Step S402, inputting the data into a plurality of different anomaly detection models;

[0100] Step S403, determining the health degree of the hard disk at the specified use time according to the outputs of the plurality of anomaly detection models.

[0101] Each step of the method shown in Figure 4 will be described in detail below.

[0102] First, in step S401, data of a plurality of indicators related to health degree of the hard disk at a specified use time is obtained.

[0103] After collecting a set of SMART data corresponding to the current usage time of the hard drive, the storage system 120 can select data on multiple health indicators related to the hard drive at the current usage time from that set of SMART data. Furthermore, after acquiring the data on these multiple indicators, the storage system 120 can also perform preprocessing on the data as described above, such as to obtain data with a uniform time distribution. It is understood that the storage system 120 is not limited to selecting multiple indicator data immediately after collecting the hard drive's SMART data to predict the hard drive's health; it can also select data on multiple indicators for a specified usage time from pre-collected SMART data of the hard drive at any time to predict the hard drive's health for that specified usage time.

[0104] In step S402, the data is input into multiple different anomaly detection models.

[0105] like Figure 2 As shown, the storage system 120 can input data of multiple indicators of the hard disk 134 selected in the previous step during a specified usage time into multiple anomaly detection models, thereby obtaining the anomaly degree of the data output by each model.

[0106] In step S403, the health of the hard drive at a specified usage time is determined based on the output of multiple anomaly detection models.

[0107] From the above references Figure 3 As can be seen from the description, by selecting training samples, the anomaly score of the hard drive output by each anomaly detection model can be correlated with the hard drive's health score. Therefore, the health score of the hard drive at a specified usage time can be determined based on the anomaly score output by each anomaly detection model at that time. For example, when the anomaly score output by the anomaly detection model is negatively correlated with the health score, the storage system 120 can directly use the anomaly score output by the anomaly detection model at that time as the hard drive's health score at that time. Alternatively, when the anomaly score output by the anomaly detection model is negatively correlated with the health score, the storage system 120 can convert the anomaly score output by the anomaly detection model into a health score or a value positively correlated with the health score based on this correlation.

[0108] In addition, the three abnormality detection models obtained by the training each have advantages and disadvantages. Specifically, the iForest abnormality detection model has the advantages of good robustness by integrating multiple binary trees, being suitable for large-scale data sets and parallel computing, and being insensitive to hyperparameters, and the disadvantage of poor accuracy for some special distribution, more abnormal samples, or high-dimensional data sets. The LOF abnormality detection model has the advantages of not making too many prior assumptions about the original data distribution and strong local anomaly discrimination, and the disadvantages of large computational complexity, not suitable for large data sets, and being too sensitive to hyperparameters. The K-Means abnormality detection model has the advantages of simple and intuitive algorithm and certain adaptability to local anomalies and all anomalies, and the disadvantage of being designed for data clustering, more suitable for data set spherical distribution scenarios, and relatively sensitive to hyperparameters.

[0109] In the embodiments of the present application, after the storage system 120 inputs the data of the multiple indicators of the hard disk at the specified use time into the multiple abnormality detection models respectively, the storage system 120 can fuse the outputs of the multiple models and obtain the health degree of the hard disk according to the fusion result, so as to balance the disadvantages of each abnormality detection model and obtain a stable and smooth health degree curve of the hard disk with respect to time.

[0110] Specifically, in the case where the output of each abnormality detection model is positively correlated with the health degree, the storage system 120 can obtain a weighted sum of the outputs of the multiple abnormality detection models as shown in formula (4), and take the result of the weighted sum as the health degree of the hard disk:

[0111]

[0112] wherein Score i is the output of each abnormality detection model, a i is the weight of each abnormality detection model, and Score is the health degree of the hard disk obtained by fusing the outputs of each abnormality detection model. It can be understood that in the case where the output of the model is negatively correlated with the health degree, Score i in formula (4) can be a value positively correlated with the health degree obtained by converting the output of each abnormality detection model.

[0113] In an embodiment, assuming that the correlation between the output of each abnormality detection model and the health degree of the hard disk is basically the same, in the above formula (4), the weights a i of each abnormality detection model can be set to be the same, so that the above formula (4) becomes the following formula (5):

[0114]

[0115] That is, the average of the outputs of multiple anomaly detection models is calculated.

[0116] In another implementation, the correlation between the output of each anomaly detection model and the hard drive health can be predetermined based on the characteristics of the model input data, such as the dimension, proportion of abnormal data, and data distribution, as well as the characteristics of each anomaly detection model. This determines the weight 'a' of each anomaly detection model in formula (4). i For example, when the distribution of SMART data from multiple usage periods on a hard drive approaches a spherical distribution, based on the characteristics of each anomaly detection model, it can be determined that the K-Means anomaly detection model can predict more accurately; therefore, the weight of the K-Means anomaly detection model can be set higher. When there is a large amount of hard drive data to be predicted, and the data has many dimensions, based on the characteristics of each anomaly detection model, it can be determined that the iForest anomaly detection model is more suitable for prediction in this situation; therefore, the weight of the iForest anomaly detection model can be set higher.

[0117] In another implementation, when making predictions for different hard drives, the weights a of each anomaly detection model in formula (4) can be dynamically adjusted based on characteristics of different hard drives, such as the proportion of abnormal data and data distribution. i To improve the accuracy of predictions.

[0118] Storage system 120 can be accessed Figure 4 The method shown determines the health status of each hard drive at various usage times. When a hard drive fails, the storage system 120 can construct a complete health curve for that hard drive from its initial use to its failure (i.e., end of life) based on its health status at various usage times, as described below. Figure 7 The health curve of the faulty hard drive is used as a benchmark health curve (or a comparison health curve) to predict the health and lifespan of other hard drives (i.e., target drives) in the future. In the following text, the faulty hard drive that provides the benchmark health curve will be referred to as the benchmark drive.

[0119] Specifically, after acquiring the health status of each benchmark disk for each usage time, the storage system 120 can further smooth the health status data using a Gaussian smoothing method, as the data may fluctuate due to measurement inaccuracies or noise during the measurement process. Specifically, the duration of the smoothing window can be set, and within the smoothing window, smoothing weights are empirically assigned to samples for each usage time. This smoothing window is then slid across the hard drive health data according to the usage time, and the health status data within the sliding window is modified based on the smoothing weights, thereby smoothing the data. Table 1 shows an example of a smoothing window.

[0120] Table 1

[0121]

[0122] As shown in Table 1, it is assumed that the length of the sliding window is set to 5 days, in which the weight of the data in the 1st day in the sliding window is set to 2.28%, the weight of the data in the 2nd day in the sliding window is set to 13.59%, the weight of the data in the 3rd day in the sliding window is set to 68.27%, and so on. As can be seen from Table 1, in the Gaussian smoothing method, the center point of each piece of data in the health curve is assumed to have the closest relationship with the smoothing result, and thus has the highest weight. As the distance from the center point gradually increases, the relationship gradually decreases, i.e., the weight gradually decreases. Subsequently, the storage system 120 can draw the health curve of the benchmark disk based on the health of the benchmark disk at each usage time after the above smoothing processing.

[0123] Figure 5 A schematic diagram of the health curve of each hard disk obtained through the above process. Figure 6 is an enlarged view of one of the health curves in Figure 5 . As shown in Figure 6 , in the health curve, the horizontal axis of the coordinates represents the time (for example, the time unit is "day") counted from the time when the hard disk is enabled, and the vertical axis of the coordinates represents the health of the hard disk. As can be seen from the plurality of health curves, the health of the hard disk obtained by the method provided by the embodiments of the present application substantially decreases stably with the smoothing of time.

[0124] After obtaining the complete health curve of each benchmark disk from the time when the hard disk is enabled to the time when the hard disk fails, the storage system 120 can add the health curves to the benchmark data set for predicting the health and life of the hard disk in use at a future time. It can be understood that the storage system 120 is not limited to obtaining the health curve of the failed hard disk included in the storage system 120 as described above, for example, the storage system 120 can receive the health curve of the failed hard disk in another storage system from the other storage system, and add the health curve to the benchmark data set.

[0125] The storage system 120 can obtain the health curve of a period of time used by the hard disk 134 in use in the storage system 120 through the same process, for example, from the time when the hard disk 134 is enabled to the current time, and the storage system 120 can predict the health and life of the hard disk 134 at a future time by comparing the partial health curve of the hard disk 134 with the benchmark health curve in the above-mentioned benchmark data set.

[0126] Figure 7 A flowchart of a method for predicting the health of a hard disk provided by the embodiments of the present application, which can be performed by the storage system 120 in Figure 2 , comprising the following steps:

[0127] In step S701, a reference health degree curve is selected from the reference data set according to the similarity of the health degree of the reference health degree curve and the health degree of the corresponding use time in the partial health degree curve of the hard disk 134 to be predicted.

[0128] In step S702, a mapping relationship between the health degree in the selected reference health degree curve and the health degree of the corresponding use time in the partial health degree curve of the hard disk 134 to be predicted is fitted.

[0129] In step S703, the health degree of the hard disk 134 at a certain future time is predicted according to the selected reference health degree curve and the mapping relationship.

[0130] The following will be described in detail Figure 7 Each step of the method shown.

[0131] First, in step S701, a reference health degree curve is selected from the reference data set according to the similarity of the health degree of the reference health degree curve and the health degree of the corresponding use time in the partial health degree curve of the hard disk 134 to be predicted.

[0132] After the storage system 120 acquires the partial health degree curve of the hard disk 134 and the plurality of reference health degree curves in the reference data set as described above, the CPU 123 can calculate the similarity of the partial health degree curve of the hard disk 134 and each reference health degree curve in the reference data set, respectively. The CPU 123 can calculate the Euclidean distance between the health degree in the partial health degree curve of the hard disk 134 and the health degree of the corresponding use time in each reference health degree curve, thereby calculating the similarity between the partial health degree curve and each reference health degree curve.

[0133] Specifically, generally, the partial health degree curve of the hard disk 134 and any reference health degree curve in the reference data set can have two similar curves, if the two similar curves are aligned in time, the health degree of the corresponding use time in the two similar curves can be the health degree of the same use time in the two similar curves.

[0134] In some practical scenarios, the two similar curves can be misaligned in time (i.e., the time length is not equal). For example, for hard disks with different capacities, the speed of decline of the health degree in time can be different, so the time span of the similar curves in the health degree curves of hard disks with different capacities is different. For example, for a hard disk with a larger capacity, because the probability of use of each storage unit in the hard disk is lower, the speed of decline of the health degree can be slower than that of a hard disk with a smaller capacity, so the time span of the similar curves in the health degree curve of the hard disk with the larger capacity is longer. To this end, the CPU 123 can determine the health degree of the corresponding use time in the two similar curves by a dynamic time warping (DTW) algorithm. Specifically, the CPU 123 shortens or extends one of the two similar curves in the time axis, so that the two similar curves are aligned in time, and the health degree of the same use time in the two similar curves after the processing is the health degree of the corresponding use time. After obtaining the health degree of the corresponding use time in the two similar curves, the CPU 123 can calculate the Euclidean distance between the health degrees of the corresponding use time, thereby calculating the similarity between the two similar curves, that is, the similarity of the health degree of the corresponding use time between the partial health degree curve of the hard disk 134 and the reference health degree curve.

[0135] After calculating the similarity between the partial health degree curve of the hard disk 134 and each reference health degree curve in the reference data set, one or more reference health degree curves with the highest similarity can be selected from the reference data set.

[0136] In step S702, a mapping relationship between the health degree in the selected reference health degree curve and the health degree of the corresponding use time in the partial health degree curve of the hard disk 134 to be predicted is fitted.

[0137] The health degree of the use time t can be obtained from the reference health degree curve, the health degree of the use time t' corresponding to the use time t can be obtained from the partial health degree curve of the hard disk 134, and the health degree of the time t and the health degree of the time t' constitute a training sample, so that a plurality of training samples corresponding to a plurality of use time pairs (t, t') can be obtained, which are used to train a regression model to fit the mapping relationship between the health degree x of the use time t in the reference health degree curve and the health degree y of the corresponding use time t' in the health degree curve of the hard disk 134. Wherein, as described above, in the case that the two similar curves in the reference health degree curve and the partial health degree curve of the hard disk 134 are time-aligned or are time-aligned after the DTW processing, the time t and the time t' are the same time. The regression model is, for example, a linear regression model with the following formula (6):

[0138] y = a + b x (6),

[0139] wherein a and b are coefficients to be determined by training samples, for example, the training of the regression model can be performed by least squares method to determine the coefficients a and b. It can be understood that in the embodiments of the present application, the regression model is not limited to a linear regression model, but can be any other form of regression model, such as a polynomial regression model.

[0140] In step S703, the health degree of the hard disk 134 at a future time is predicted according to the selected benchmark health degree curve and the mapping relationship.

[0141] After fitting the mapping relationship between the health degree in the partial health degree curve of the hard disk 134 and the health degree in the corresponding use time in the benchmark health degree curve, the health degree of the hard disk 134 at a future time t1 can be predicted based on the mapping relationship and the benchmark health degree curve. Specifically, in order to predict the health degree y1 of the hard disk 134 at a future time t1, the health degree x1 of the use time t2 corresponding to the time t1 in the benchmark health degree curve can be obtained, and x1 is substituted into formula (6), so that the health degree y1 of the hard disk 134 can be predicted. Wherein, similar to the above, in the case that the partial health degree curve of the hard disk 134 and the two similar curves in the benchmark health degree curve are time-aligned, the above-mentioned time t1 and t2 can be the same time, and in the case that the time of the two similar curves is not aligned, the time t2 corresponding to the time t1 can be determined by the DTW method.

[0142] In the case of determining a plurality of (for example, m) benchmark health degree curves in the foregoing steps, m health degrees y i (where i is 1 to m) can be weighted and summed as shown in formula (7) to obtain the health degree Y1 of the hard disk 134: i

[0143]

[0144] wherein k i is a preset weight corresponding to each benchmark health degree curve, for example, the weights corresponding to each benchmark health degree curve can be determined according to the size of the similarity of each benchmark health degree curve and the partial health degree curve of the hard disk 134.

[0145] As can be seen from the above prediction process, the regression model trained as described above can migrate the process of the health degree in the benchmark health degree curve changing with time to the health degree curve of the target disk (i.e. the hard disk 134), so that the regression model plays a role of migrating knowledge, which can also be called a migration model. ​

[0146] Storage system 120 can be accessed Figure 7 The method shown predicts the health of hard disk 134 at multiple points in the future. For example, it can predict the health of hard disk 134 each day in the future, thereby predicting a future health curve for hard disk 134. Storage system 120 can preset a health threshold for hard disk 134, which corresponds to the health of hard disk 134 at the time of failure. Therefore, the storage system can determine the time when the health of hard disk 134 reaches the threshold from the predicted future health curve of hard disk 134, and determine the remaining lifespan of the hard disk based on that time. Figure 8 This is a schematic diagram of the predicted hard disk health curve provided in an embodiment of this application. Figure 8 As shown in the diagram, the horizontal axis represents the time elapsed since the hard drive was enabled, and the vertical axis represents the hard drive's health. Assuming... Figure 8 The dotted line connecting the lower and middle sections represents the health curve C1 of the benchmark disk, while the dotted line connecting the upper sections represents the health curve C2 of the hard drive 134 to be predicted. The solid line portion of curve C2 represents the health curve determined based on SMART data collected over a period of time after the hard drive 134 was enabled. The dashed line portion of curve C2 represents the health curve obtained through… Figure 7 The method shown predicts the health curve of hard disk 134 at a future time. Specifically, for example, in order to predict the health of time t1 in curve C2, the time t2 corresponding to time t1 in curve C1 can be determined first, and the health of time t2 in curve C1 can be obtained as x1. Substituting x1 into the above formula (6), the health of time t1 in curve C2 as y1 can be calculated. When predicting the health of hard disk 134 based on multiple selected benchmark health curves, the health of hard disk 134 as Y1 can also be calculated using the above formula (7).

[0147] like Figure 8 As shown, assuming a health threshold is set in curve C2, which corresponds to the end of the lifespan of the hard drive 134 to be predicted, then after predicting the future health curve of the hard drive 134 as described above, the time t3 corresponding to the threshold can be determined in curve C1, and the time t3 is regarded as the end of the lifespan of the hard drive 134.

[0148] Understandable. Figure 7 The method shown is merely one implementation of the method for predicting the health and lifespan of a hard drive in the future, and the embodiments of this application are not limited thereto. For example, in another implementation, after selecting a benchmark health curve corresponding to the hard drive 134 to be predicted, the storage system 120 can directly use the health of the hard drive 134 at time t1 corresponding to time t2 in the benchmark health curve as the health of the hard drive 134 at the future time t1 when predicting the health of the hard drive 134.

[0149] Figure 9 The architecture diagram of the storage device provided by the embodiments of the present application can be used to execute any method shown in Figure 3 、 Figure 4 or Figure 7 The storage device comprises:

[0150] An acquisition unit 91 is configured to acquire data of a plurality of indexes related to health degree of a hard disk at a specified use time;

[0151] An input unit 92 is configured to input the data into a plurality of different models;

[0152] A determination unit 93 is configured to determine the health degree of the hard disk at the specified use time according to outputs of the plurality of models.

[0153] In an embodiment, the determination unit 93 is specifically configured to determine the health degree of the hard disk at the specified use time based on a weighted sum of the outputs of the plurality of models.

[0154] In an embodiment, each model of the plurality of models is trained based on an anomaly detection algorithm, and the anomaly detection algorithms adopted by the plurality of models are different.

[0155] In an embodiment, the number of the plurality of models is three, and the anomaly detection algorithms adopted by the three models are isolated forest algorithm, local outlier factor algorithm and K-means clustering algorithm respectively.

[0156] In an embodiment, the plurality of models are sent to the storage device by a training device or are trained by the storage device, and the training device is configured to train the plurality of models.

[0157] In an embodiment, the plurality of models are trained by sampling data of a plurality of indexes related to health degree of a failed hard disk within a preset time period before the end of the life of the failed hard disk.

[0158] In an embodiment, the hard disk comprises a target disk, and the storage device further comprises:

[0159] An acquisition or generation unit is configured to acquire or generate a first data set of a plurality of benchmark disks, and the first data set comprises health degrees of the plurality of benchmark disks at a plurality of use times;

[0160] A generation unit is configured to generate a second data set of the target disk, and the second data set comprises health degrees of the target disk at a plurality of use times, and a time span of the plurality of use times in the first data set is greater than a time span of the plurality of use times in the second data set;

[0161] The selecting unit is configured to select the pair of disks according to the similarity of the health degrees of the plurality of use times corresponding to the first data set and the second data set.

[0162] The predicting unit is configured to predict the health degree of the target disk at the specified time in the future according to the selected first data set of the pair of disks.

[0163] In an embodiment, the predicting unit is specifically configured to: fit a mapping relationship between the health degrees of the first use time of the selected pair of disks and the health degrees of the second use time of the target disk, the first use time and the second use time being corresponding times; and predict the health degrees of the target disk at the plurality of specified times in the future according to the mapping relationship and the first data set.

[0164] In an embodiment, the determining unit 93 is further configured to determine, according to the predicted health degrees of the target disk at the plurality of specified times in the future, a time at which the health degree of the target disk reaches a threshold value, and take the time at which the health degree reaches the threshold value as the end-of-life time of the target disk.

[0165] The third aspect of the present application provides a storage device, comprising a processor and a memory, the memory storing executable computer program instructions, and the processor executes the executable computer program instructions to execute any method shown in Figure 3 、 Figure 4 or Figure 7 .

[0166] The fourth aspect of the present application provides a computer readable storage medium, characterized in that the computer readable storage medium stores computer program instructions, when the computer program instructions are executed in a computer or a processor, the computer or the processor executes any method shown in Figure 3 、 Figure 4 or Figure 7 .

[0167] The fifth aspect of the present application provides a computer program product, comprising computer program instructions, when the computer program instructions are executed in a computer or a processor, the computer or the processor executes the method of the first aspect or the possible implementation manners of the first aspect.

[0168] It should be understood that the "first", "second", and the like descriptions herein are only for the sake of simple description of similar concepts, and do not have other limiting effects.

[0169] Those skilled in the art can clearly understand that the descriptions of various embodiments provided in the present application can be mutually referred to, and for the convenience and brevity of description, for example, the functions of various devices and the steps performed by the devices provided in the embodiments of the present application can refer to the related descriptions of the method embodiments of the present application, and the method embodiments and the device embodiments can also be mutually referred to.

[0170] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media.

[0171] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways without exceeding the scope of the present application. For example, the above-described embodiments are merely illustrative, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. The units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected to achieve the purpose of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0172] In addition, the illustrative examples described above and the accompanying examples can be combined or integrated with each other. Moreover, those skilled in the art will recognize various modifications and changes that can be made to the above-described illustrative examples and that such modifications and changes are also within the scope of this application. Although the above description has been made with respect to particular examples and embodiments, it will be understood that various changes and modifications can be made without departing from the scope of the application. It is therefore intended that the application not be limited to the illustrative examples and embodiments described herein, but that the application include all embodiments falling within the scope of the appended claims.

[0173] The above description is intended to enable any person skilled in the art to make and use the application. Various modifications to the embodiments described above will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without the use of the inventive faculty. Nothing in the above description of examples should be interpreted as a limitation on the scope of the application. Accordingly, the application is to be construed as including all embodiments falling within the scope of the claims, and equivalents thereof.

Claims

1. A method for hard disk health assessment, the method comprising: The method is executed by a storage device, comprising: obtaining data of a plurality of indexes related to health degree of a hard disk in a specified use time; inputting the data into a plurality of different models; the number of the plurality of models is three, and the three models adopt anomaly detection algorithms respectively as an isolated forest algorithm, a local outlier factor algorithm and a K-means clustering algorithm; the isolated forest algorithm is used for training an iforest anomaly detection model, and the iforest anomaly detection model is used for determining a layer height expectation value of the data in the isolated forest model; the smaller the layer height expectation value is, the higher the abnormality degree of the hard disk is; the local outlier factor algorithm is used for training an LOF anomaly detection model, and the LOF anomaly detection model is used for calculating an LOF score of the data; the LOF score greater than 1 represents that the abnormality degree of the hard disk is higher, and the LOF score less than or equal to 1 represents that the abnormality degree of the hard disk is lower; the K-means clustering algorithm is used for training a K-means anomaly detection model, and the K-means anomaly detection model is used for calculating a distance between the data and a centroid point; the greater the distance is, the higher the abnormality degree of the hard disk is; the centroid point is obtained by clustering a plurality of samples by a clustering algorithm; the advantages and disadvantages of the plurality of models are complementary; determining the health degree of the hard disk in the specified use time according to outputs of the plurality of models, so as to balance the disadvantages of the respective models and obtain a change curve of the health degree of the hard disk relative to time.

2. The method of claim 1, wherein, The determination of the health degree of the hard disk in the specified use time according to the outputs of the plurality of models specifically comprises determining the health degree of the hard disk in the specified use time based on a weighted sum of the outputs of the plurality of models.

3. The method according to claim 1 or 2, characterized in that, Each of the plurality of models is trained based on an anomaly detection algorithm, and the anomaly detection algorithms adopted by the respective models are different.

4. The method according to any one of claims 1 to 3, characterized in that, The plurality of models are sent to the storage device by a training device or are obtained by the storage device, and the training device is used for training the plurality of models.

5. The method of claim 4, wherein, The plurality of models are trained by sampling data, which is sampling data of a plurality of indexes related to health degree of a fault hard disk in a preset use period before the end of the life of the fault hard disk.

6. The method of any of claims 1-5, the hard disk comprising a target disk, characterized in that, The method further comprises: obtaining or generating a first data set of a plurality of benchmark disks, the first data set comprising health degrees of the benchmark disks in a plurality of use times; generating a second data set of the target disk, the second data set comprising health degrees of the target disk in a plurality of use times, and a time span of the plurality of use times in the first data set is greater than a time span of the plurality of use times in the second data set; selecting a benchmark disk according to the similarity of the health degrees of the plurality of use times corresponding to the first data set and the second data set; predicting the health degree of the target disk at a specified time in the future according to the first data set of the selected benchmark disk.

7. The method of claim 6, wherein, The prediction of the health degree of the target disk at the specified time in the future according to the first data set of the selected benchmark disk comprises: fitting a mapping relationship between the health degree of the selected first use time of the reference disk and the health degree of the second use time of the target disk, the first use time and the second use time being corresponding times; and predicting the health degrees of the target disk at a plurality of specified future times according to the mapping relationship and the first data set.

8. The method of claim 7, wherein, The method further includes determining a time when the health degree of the target disk reaches a threshold value according to the predicted health degrees of the target disk at the plurality of specified future times, and taking the time when the health degree reaches the threshold value as an end-of-life time of the target disk.

9. A storage device, comprising: The storage device includes: an acquisition unit configured to acquire data of a plurality of indexes related to the health degree of the hard disk at a specified use time; an input unit configured to input the data into a plurality of different models; the number of the plurality of models is three, and the three models employ anomaly detection algorithms respectively, the anomaly detection algorithms being an isolation forest algorithm, a local outlier factor algorithm and a K-means clustering algorithm; the isolation forest algorithm is used to train an iforest anomaly detection model, the iforest anomaly detection model is used to determine a layer height expectation value of the data in the isolation forest model; the smaller the layer height expectation value, the higher the abnormality degree of the hard disk; the local outlier factor algorithm is used to train a LOF anomaly detection model, the LOF anomaly detection model is used to calculate a LOF score of the data; the LOF score greater than 1 represents a higher abnormality degree of the hard disk, and the LOF score less than or equal to 1 represents a lower abnormality degree of the hard disk; the K-means clustering algorithm is used to train a K-means anomaly detection model, the K-means anomaly detection model is used to calculate a distance between the data and a centroid point, and the greater the distance, the higher the abnormality degree of the hard disk; the centroid point is obtained by clustering a plurality of samples by a clustering algorithm; the advantages and disadvantages of the plurality of models are complementary; a determination unit configured to determine the health degree of the hard disk at the specified use time according to outputs of the plurality of models, so as to balance the disadvantages of the respective models and obtain a change curve of the health degree of the hard disk with respect to time.

10. The storage device of claim 9, wherein, The determination unit is specifically configured to determine the health degree of the hard disk at the specified use time based on a weighted sum of the outputs of the plurality of models.

11. The storage device of claim 9 or 10, wherein, Each of the plurality of models is trained based on an anomaly detection algorithm, and the anomaly detection algorithms employed by the respective models are different.

12. The storage device of any of claims 9-11, wherein, The plurality of models are sent to the storage device by a training device or are trained by the storage device, and the training device is configured to train the plurality of models.

13. The storage device of claim 12, wherein, The plurality of models are trained by sampling data, the sampling data being a plurality of indexes related to the health degree of a failed hard disk within a preset use period before the end of the life of the hard disk.

14. The storage device of any of claims 9-13, the hard disk comprising a target disk, characterized in that, The storage device further includes: an acquisition or generation unit configured to acquire or generate a plurality of first data sets of reference disks, the first data sets including health degrees of the reference disks at a plurality of use times; The generating unit is configured to generate a second data set of the target disk, the second data set comprising health degrees of the target disk at a plurality of usage times, a time span of the plurality of usage times in the first data set being greater than a time span of the plurality of usage times in the second data set; The selecting unit is configured to select a benchmark disk according to a similarity of the health degrees of the plurality of usage times corresponding to the first data set and the second data set; The predicting unit is configured to predict a health degree of the target disk at a specified future time according to the first data set of the selected benchmark disk.

15. The storage device of claim 14, wherein, The predicting unit is specifically configured to: fit a mapping relationship between the health degrees of the first usage times of the selected benchmark disk and the second usage times of the target disk, the first usage times and the second usage times being corresponding times; and predict the health degrees of the target disk at a plurality of specified future times according to the mapping relationship and the first data set.

16. The storage device of claim 15, wherein, The determining unit is further configured to determine, according to the predicted health degrees of the target disk at the plurality of specified future times, a time at which the health degree of the target disk reaches a threshold value, and take the time at which the health degree reaches the threshold value as an end-of-life time of the target disk.

17. A storage device, comprising: A processor and a memory are included, the memory storing executable computer program instructions, and the processor executes the executable computer program instructions to implement the method of any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions, when the computer program instructions are executed in a computer or a processor, the computer or the processor executes the method of any one of claims 1-8.

Images