Use of recovery behavior for prognosis and in-situ repair of data storage devices

Abstract

The present invention discloses a method for identifying a problematic data storage device such as a hard disk drive (HDD) in a data storage system, the method involving retrieving and evaluating a corresponding recovery log, such as a media error portion of a device status log, from each of a plurality of HDDs. Based on each recovery log, a value of a full recovery per hour (FRPH) metric is determined for each read / write head of each corresponding HDD. Generally speaking, the FRPH metric characterizes the amount of time that a head has spent performing a recovery operation. In response to a specific head FRPH reaching a predetermined threshold, an in-situ repair of the HDD in which the specific head is operating is determined. Similarly, in the context of a solid-state drive (SSD), a delay metric can be determined based on the time spent waiting to resolve an input / output (IO) request conflict, and the in-situ repair can be based on the delay metric.

Description

Technical Field

[0001] Embodiments of the present invention may relate generally to data storage systems, and more particularly to prognosis and in-situ repair of data storage devices. Background Art

[0002] There is a commercial need for high capacity digital data storage systems in which multiple data storage devices (DSDs) are housed in a common housing. The data storage system typically includes a large housing housing multiple shelves on which rows of DSDs are mounted. A typical method of using a data storage system is to remove or remove from service DSDs that fail or fail in the field. Such DSDs are then typically replaced with replacement DSDs by a technician at an in-house laboratory or customer deployment site. The failed drives are either destroyed by the customer or returned to the manufacturer for failure analysis.

[0003] Any approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Embodiments are illustrated by way of example and not limitation in the accompanying drawings in which like reference numerals refer to similar elements and in which:

[0005] Figure 1A is a plan view showing a hard disk drive (HDD) according to one embodiment;

[0006] Figure 1B is a block diagram illustrating a solid state drive (SSD) according to one embodiment;

[0007] Figure 2 is a perspective view showing a data storage system according to one embodiment;

[0008] Figure 3 is a block diagram illustrating a data storage system architecture according to one embodiment; and

[0009] Figure 4 is a flow chart illustrating a method of identifying a problematic hard drive, according to one embodiment. DETAILED DESCRIPTION

[0010] The present invention describes methods for prognosis and in-situ repair of data storage devices. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the embodiments of the present invention described herein. However, it will be apparent that the embodiments of the present invention described herein can be practiced without these specific details. In other cases, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the embodiments of the present invention described herein.

[0011] Physical Description of an Exemplary Operational Context Data Storage System

[0012] There is a commercial need for a high capacity digital data storage system in which multiple data storage devices (DSDs) such as hard disk drives (HDDs) and / or solid state drives (SSDs) are housed in a common housing. The data storage system typically includes a large housing housing a plurality of shelves on which rows of DSDs are mounted. Figure 2 2 is a perspective view showing an example of a data storage system according to an embodiment. The data storage system 200 may include a system housing 202 (or "rack 202") in which a plurality of data storage system trays 204 are housed. For example, each tray 204 may be placed or slid into a corresponding slot within the rack 202. The rack 202 also houses a system controller and may also house switches, storage servers, application servers, power supplies, cooling fans, etc.

[0013] Figure 3is a block diagram illustrating a data storage system architecture according to one embodiment. The exemplary architecture 300 illustrates a data storage system 200 including a plurality of hard disk drives (HDDs) 304a (HDD1), 304b (HDD2), and 304n (HDDn), where n represents any number of HDDs (and / or SSDs) that may vary depending on the specific implementation. Each HDD 304a-304n communicates with a data storage system controller 312 via a communication interface 322 according to a corresponding communication protocol 323 and is under the control of the data storage system controller. Each HDD 304a-304n includes a corresponding non-volatile memory (NVM) 306 (e.g., in the case of a HDD, typically in the form of a rotating disk medium), which is controlled by a corresponding HDD controller 308 including a log module 310. Each log module 310 (which may be, for example, a component of a corresponding system-on-chip (SoC), such as a portion of a data channel circuit or a hard disk controller circuit, as shown in the context of a HDD) is capable of recording actions taken by the HDD controller 308, such as recording read, write and seek errors, associated recovery actions, and other recent and long-term HDD state information. The system controller 312 of the data storage system 200 includes a memory 313, a processor 315, and a reliability engine 314 and an associated FRPH (Full Recovery Per Hour) module 316, which together represent process functions related to prognosis and in-situ repair of the HDDs 304a-304n, as described in more detail elsewhere herein.

[0014] Processes, functions, procedures, actions, method steps, etc. described herein as being performed or executable by the system controller 312 or the HDDs 304a-304n may include being performed by executing one or more sequences of instructions stored in one or more memory units, and when executed by one or more processors, result in such performance. The system controller 312 may be implemented in any form and / or combination of software, hardware, and firmware. For example, and in accordance with one embodiment, the system controller 312 includes an application specific integrated circuit (ASIC) including at least one memory unit for storing such instructions (such as firmware as a non-limiting example) and at least one processor for executing such instructions, enabling prognosis of potentially problematic HDDs with respect to the plurality of HDDs 304a-304n (e.g., "silent" failed drives, which may be struggling to complete data transactions internally but have not yet disclosed a failure), and managing, controlling, and facilitating in-situ repair actions corresponding to failed and failing HDDs.

[0015] The data storage system 200 may be communicatively coupled to a host 350, which may be implemented as a hardware machine (for non-limiting examples, a computer or hardware server, etc.) on which executable code is executed, or as software instructions executable by one or more processors (for non-limiting examples, a software server such as a database server, an application server, a media server, etc.). The host 350 generally represents a client of the data storage system 200 and has the ability to make read and write requests (input / output or "IO") to the data storage system 200. It is noted that the system controller 312 may also be referred to as a "host" because the term is generally used with reference to any device that makes IO calls to a data storage device or array of devices, such as HDDs 304a-304n. Background Art

[0017] As used herein, the term "substantially" should be understood to describe a feature that is mostly or nearly structured, configured, dimensioned, etc., but in practice manufacturing tolerances, etc. cause the structure, configuration, dimensions, etc. not always or necessarily to be exactly as described. For example, describing a structure as "substantially vertical" would give the term its ordinary meaning, such that the sidewalls are vertical for all practical purposes, but may not be at exactly 90 degrees.

[0018] While terms such as "best," "optimize," "minimum," "minimize," etc. may not have certain values ​​associated with them, if such terms are used herein, it is intended that one of ordinary skill in the art will understand that such terms will include affecting values, parameters, metrics, etc. in beneficial directions consistent with the present disclosure as a whole. For example, describing a value of something as "minimum" does not require that the value actually equal some theoretical minimum value (e.g., zero), but should be understood in a practical sense that the corresponding goal is to move the value in a beneficial direction toward the theoretical minimum value.

[0019] Some data storage systems implement diagnostic and in-situ (e.g., "in place" or "in place") repair procedures. This means that system HDDs that have failed during field operation are diagnosed and repaired to the extent possible on site, and if they cannot be repaired, they are deactivated in situ rather than replaced. Preferably, in-situ repair requires an automated or semi-automated process, and the HDD in question remains in the system housing. Many HDD repairs do not result in any loss of capacity or performance. HDDs can be repaired in a way that still allows system software to continue to be used, although with reduced capacity or performance. Unrepairable HDDs may be allowed to fail in place within the system, such as in the case where the system comes with over-configured storage capacity, which allows the availability of the full specified storage capacity despite the presence of many drive failures. In-situ repairs can eliminate the cost of dispatching service technicians to replace drives that fail in the field, as well as the cost of these replacement drives. In addition, in-situ repairs can reduce the AFR (annual failure rate) of storage devices, and will likely eliminate the ARR (annual repair rate) when future product releases adopt a "field failure" model.

[0020] Implementation of in-situ repair in HDDs can minimize the amount of over-provisioning necessary in data storage systems to be able to provide lifetime hardware guarantees. As mentioned, some storage systems (and / or storage system operating system software) may be able to continue using many HDDs after they have failed. Analysis of failed drives has shown that many HDDs that fail in the field can be restored to full or partial use with relatively simple techniques.

[0021] Reliability Engine

[0022] refer to Figure 3 According to one embodiment, a software component referred to as reliability engine 314 is responsible for performing or managing HDD diagnostic, prognostic, and repair operations. With respect to diagnostics, when a system (e.g., system controller 312 of data storage system 200) detects a problem with a HDD 304a-304n of data storage system 200, the system calls reliability engine 314 and passes information describing the problem the system has with the HDD.

[0023] Typically, the reliability engine 314 can enter a diagnostic state regarding the HDD and analyze the HDD to identify the root cause of the problem. With respect to repair, once the root cause of the problem is identified, the reliability engine 314 notifies the system controller 312 of the logic of the recommended repair action, which can schedule the recommended repair action according to the urgency level based on system events (e.g., code upload, impending shutdown / restart) and the disk redundancy level within the system. Once the system controller 312 requests a repair, the HDD is typically offline and the reliability engine 314 performs the repair (which may take anywhere from a few seconds to a few days). At the end of the repair and post-repair diagnostics, the HDD can be restored to full use, restored to use at reduced capacity and / or performance, or the HDD can be disabled from the system.

[0024] According to an embodiment, regarding prognosis, the reliability engine 314 also periodically monitors the health of each HDD when each HDD is online and used by the system. This can be done by reading and analyzing a limited time subset of the internal E6 log 317 of the HDD, such as in conjunction with the FRPH module 316 or with the help of the FRPH module, as described in more detail elsewhere herein. Therefore, the reliability engine 314 can actively detect, for example, a delay problem or an error rate problem in the HDD, which affects the host visible performance, but is not serious enough for the system to cause the HDD to fail. For example, a HDD that consistently enters a high DRP (data recovery process) level when accessing a specific head or area of ​​the medium will generally exhibit a higher IO completion delay than its peer drive. In erasure coding or RAID-based systems of some storage systems, the host visible delay is generally limited by the performance of the slowest HDD in the data set. The reliability engine 314 can asynchronously notify the system controller 312 of the logic of the recommended repair operation of the HDD. Once the system allows the reliability engine 314 to continue repairing to remove the high-latency head / surface, the host visible delay degradation is resolved.

[0025] Hard Drive Prognosis Tool: Full Recovery in Hours

[0026] Figure 4 is a flow chart illustrating a method of identifying a problematic hard drive, according to one embodiment. Figure 4 The process or flow may be implemented as one or more instruction sequences stored in one or more memory units and, when executed by one or more processors, cause the process to be performed. For example, a program stored in one or more memory units (e.g., a ROM inherent to firmware) and, when executed by one or more processors (e.g., Figure 3 When the processor 315 of the system controller 312 executes Figure 4The sequence of instructions for the process shown (eg, reliability engine 314 embodied in firmware of system controller 312 of FIG. 1 ) is shown.

[0027] Additionally, a similar approach is contemplated for identifying problematic solid-state drives (SSDs), with appropriate modifications based on the different technologies underlying hard disk drives and solid-state drives. For example, for HDDs, the primary reason that the DRP step increases host-visible latency is due to rotational latency due to having to wait for an additional rotation of the media to retry the operation. In contrast, in the context of SSDs, there is no rotational latency, and thus a key cause of host-visible latency is scenarios where host IO must wait for other activities being performed by the SSD, such as host IO requests conflicting with background tasks (e.g., garbage collection, cell refresh, read disturb, etc.), host IO requests conflicting with other host IO requests, and increased latency due to non-volatile memory components requiring longer program / erase operations as they exhibit degradation due to aging and / or wear. Thus, in addition to SSDs, a metric functionally equivalent to the FRPH function can be used to quantify the variability of latency (and, likewise, the variability of QoS (quality of service)) in storage system SSDs.

[0028] At block 402, a corresponding recovery log is retrieved from each of a plurality of hard disk drives. Figure 2 and Figure 3 ) of the system controller 312 ( Figure 3 ) from a plurality of HDDs 304a-304n ( Figure 3 ) of one or more log modules 310 ( Figure 3 ) Retrieve E6 log 317 ( Figure 3). The E6 log 317 refers to a specific implementation of a binary file used to snapshot the state of the HDD. The E6 log 317 contains a large number (tens of) sub-segments, including a media error log, and is typically in the size range (order of magnitude) of 10MB to 100MB, depending on which "modes" (segment groups) are acquired. There is also an incremental E6 log capability that only acquires "new" state, that is, since the most recent full or incremental acquisition. An incremental E6 log can be as small as a few MB, but can be up to the size of the full E6 log 317. The segments of the E6 log 317 have various themes / purposes, some are globally fixed-size "drive life" states, and some grow over time as more drive activity accumulates. Some of the data elements in a segment are retained during the life of the drive, and some are wrapped because the HDD autonomously deletes older states to avoid exceeding a bounded upper limit on the size. According to one embodiment, the media error log portion of the E6 log 317 is used below to calculate the following FRPH (full recoveries per hour) metric and is functionally referred to as a "recovery log" because this particular portion or segment of the E6 log 317 records the data recovery steps (e.g., seek recovery, read recovery, write recovery) performed by each read / write head within the corresponding HDD.

[0029] At box 404, a value of a metric referred to as FRPH is determined for each respective read-write head of each respective HDD in the plurality of HDDs based on each respective recovery log. For example, as described below, FRPH values ​​are calculated for each read-write head (e.g., read-write head 110a of FIG. 1 ) of each of one or more of the plurality of HDDs 304a-304n based on the recovery log retrieved at box 402. A DRP or recovery "step" is an action that a HDD dynamically invokes to recover data from one or more sectors that are unreadable after executing a previous step. For example, there may be hundreds of possible DRP steps available to a HDD / head, which may be more generally categorized in one of the categories for read, write, and seek recovery steps, where higher step numbers represent more involved recovery algorithms, and therefore take longer to execute than lower numbered DRP steps.

[0030] According to one embodiment, "full recovery" is the execution of all possible DRP steps. If full recovery is not successful, the host receives a hard error indicating that data cannot be transferred.

[0031] At block 404a, the amount of time spent on all recovery steps performed by the head is summed over a particular length of time (i.e., the evaluation period). For example, the relevant recovery log is analyzed, and for a non-limiting example, the severity (time spent according to each "type" or "level" of recovery steps) of all recovery steps performed by each HDD head (e.g., the HDD media error log section from the E6 log 317) over the last 300 hours of operating time is summed. According to one embodiment, the evaluation period is a configurable parameter that allows an arbitrary time window over which recovery steps are analyzed.

[0032] Conceptually, a full recovery means that all recovery steps of a recovery event have been run, and the more recovery steps required, the more serious the problem is and the longer it will take for data recovery. However, according to one embodiment, for algorithmic purposes, a "full recovery" is characterized as corresponding to a predetermined length of time that the head spends on one or more recovery steps. According to one embodiment, a full recovery is established as the head actually spending about 2.5 seconds on one or more recovery steps. However, the manner in which a full recovery is characterized for algorithmic purposes may vary from implementation to implementation. Therefore, for each predetermined amount of time (e.g., 2.5 seconds of real time) that the head spends on a recovery step, a "full recovery" is counted. With respect to the severity of the recovery steps, as described with reference to the exemplary system, there are hundreds of possible DRP recovery steps, where higher step numbers represent more involved complex recovery algorithms, and therefore, take longer to execute than lower numbered DRP steps. In addition, recovery steps are classified into groups of similar complexity and therefore have similar execution times. For example, one thousand "step 4" recovery steps may be equivalent in time to one hundred "step 8" recovery steps.

[0033] Therefore, at box 404b, an equal number of full recoveries within a specific duration is determined. Referring to the aforementioned example, the number of 2.5 second time units spent on the recovery step by the head within the 300-hour evaluation period establishes the number of full recoveries for the head. Then, at box 404c, based on the number of full recoveries generated and the specific duration evaluated, the FRPH value of the head is determined, which represents the DRP recovery time spent per head on average per hour (e.g., in units of the time spent for the maximum length full recovery), for example, the DRP recovery time spent per hour on average over the past 300 hours. Therefore, at box 404, boxes 404a-404c are iterated for each head from each HDD 304a-304n to be evaluated. In addition, FRPH is calculated by checking the recovery log to identify each IO that requires any recovery step, and then accumulating an estimated value of the total amount of time to perform all recovery steps representing the use of the IO, and the FRPH metric is calculated separately for each head in the HDD.

[0034] The calculation of the FRPH metric at box 404 can be achieved by analyzing the recovery log of the evaluation cycle, summing the total time spent by each head in each of the hundreds of DRP recovery steps, which will effectively produce a histogram recording the number of calls to each recovery step / level by each head. For each recovery step, a constant time value that estimates how long the specific recovery step is executed can be multiplied by the corresponding call of the recovery step, which will produce the total amount of time spent by each head in any recovery step. The total amount of time spent in all recovery steps can be divided by the evaluation cycle (e.g., 300 hours), which will obtain the average amount of time spent per hour in any recovery step for each head. The average time / hour spent in all recovery steps can be normalized by dividing the time established for full recovery (e.g., the equivalent of 2.5 seconds of "full recovery"), which will produce a FRPH value normalized to the evaluation cycle, wherein a FRPH value of one will mean that on average per hour (e.g., in the past 300 hours of the evaluation cycle) the head is used to perform a full recovery in the recovery operation mode.

[0035] At block 406, in response to a particular head FRPH reaching a predetermined FRPH threshold, an in-situ repair of the HDD in which the particular head operates is determined. For example, and in accordance with one embodiment, an HDD having one or more heads having a FRPH value of one (e.g., a threshold) or greater for a particular length of time (e.g., a 300 hour evaluation period) is identified as a problematic HDD, i.e., one that may be failing "silently." Additionally, the FRPH metric may be made available for querying by a client, such as via a RESTful API (representational state transfer application program interface), whereby the client will decide whether the FRPH value warrants a mitigation action, and if so, will need to initiate such action itself.

[0036] According to one embodiment, if any HDD 304a-304n has one or more heads with a FRPH greater than or equal to one, the head with the highest FRPH in the problematic HDD is identified and a "head ejection" (head detachment) is performed on the head, that is, the head is classified (e.g., classified to the system controller 312 and / or classified to the HDD controller 308) as not eligible for use in further IO operations, thereby effectively disabling further use / operation. Alternatively, according to one embodiment, the originally problematic HDD may be reformatted to its current capacity. For example, in the case where one or more heads have been "ejected" in a previous repair operation and it is decided not to eject another head at this time, the HDD may be reformatted to its current capacity taking into account the reduced capacity corresponding to the previously ejected head. In addition, and according to one embodiment, because it can be determined from the media error log how much area of ​​the media and which problematic areas contribute to the FRPH value, another repair option involves forcibly leaving out the problematic area of ​​the media (e.g., marking the area as unavailable for further IO) instead of performing a longer and more complex reformat or eject repair.

[0037] Intelligent system-level reliability solutions

[0038] Historical storage systems may include storage devices with autonomous, independently designed diagnostic and device reliability schemes, which may result in reduced efficiency and effectiveness of the overall storage system. In contrast, an expanded holistic view of operational functionality at and between different storage devices distributed throughout the storage system may result in a more efficient and reliable overall system.

[0039] Data storage devices, such as those constituting the data storage system 200 ( Figure 2 and Figure 3 ) of HDD304a-304n( Figure 3 ), which can be considered as sensitive sensors of factors that may affect the data integrity and reliability associated with each DSD, such as temperature, airflow, vibration, etc. in the context of HDDs, and age, storage cycles, etc. in the context of SSDs. In addition, HDDs 304a-304n can be a source of information regarding position error signals (PES) corresponding to specific media heads or regions and factors contributing thereto, DRP recovery operations, etc. Therefore, when used in conjunction with controllers such as controller 312 ( Figure 3), the data integrity and recovery information that each DSD may know internally and their combination can provide an inexpensive but valuable extended vision of data degradation symptoms at the device level to serve as an informed and effective diagnostic, prognostic, and repair control point at the system level. There may also be additional information about the DSDs available at the system level (e.g., available to the system controller 312) that can be facilitated and / or combined to facilitate intelligent predictive decisions at the system level. For example, in the context of HDDs, the system controller may be made aware of information such as the storage device form factor, the number of disks (and / or actuators), the RPM (revolutions per minute) of the disks, and other common variations in HDD models, as well as the location of each storage device in a particular shelf and system rack (e.g., Figure 2 2, relative proximity to fans, fan speeds, relative proximity to heat sources, and other factors external to each device itself. It is noted that the HDDs 304a-304n and the system controller 312 may be programmed to "learn" over time. Therefore, the factors that may affect the reliability of data associated with each HDD may not be static, that is, these factors (and repair decisions based on these factors) may be dynamic over time.

[0040] The closed-loop nature of the HDD monitoring diagnosis / prognosis repair described herein is capable of continuously monitoring the HDD log, showing the calculated metrics of the behavior of this one HDD, and then comparing the metrics of each HDD with the metrics of its peer drive to determine "pass / fail". It should be noted that the event log of each HDD is used to calculate the metrics of each HDD, where each HDD is only aware of itself. Here, the system can see the values ​​of the metrics of many HDDs, and compare the metrics of one HDD with the metrics of the HDD peer drive, where the definition of the peer drive can be variable, for example, by similar model types, firmware revisions, and even by the use case of the HDD in the rack. Similarly, the calculation of each FRPH metric only evaluates the health of the head on one drive. However, the diagnostic host can set a threshold for which value of the FRPH metric constitutes a pass or fail by the following operation: consider the degree of abnormality of the FRPH metric of each drive relative to the larger ecosystem visible to the diagnostic host. For example and according to one embodiment, the pass / fail threshold (i.e., FRPH threshold) can be set to a value equal to two or three standard deviations, which is higher than the average (or median) FRPH score of the rest of the deployment.

[0041] Hard Drive Configuration

[0042] As discussed, embodiments may be used in the context of a data storage system that employs multiple data storage devices (DSDs), such as HDDs. Thus, according to one embodiment, Figure 1A A plan view of HDD 100 is shown in FIG. 1 to illustrate exemplary operating components.

[0043] Figure 1A A functional arrangement of components of an HDD 100 including a slider 110b is shown, the slider 110b including a magnetic read-write head 110a. The slider 110b and the head 110a may be collectively referred to as a head slider. The HDD 100 includes at least one head gimbal assembly (HGA) 110 having a head slider, a lead suspension 110c attached to the head slider, typically via a flexure, and a load beam 110d attached to the lead suspension 110c. The HDD 100 also includes at least one recording medium 120 rotatably mounted on a spindle 124 and a drive motor (not visible) attached to the spindle 124 for rotating the medium 120. The read-write head 110a (which may also be referred to as a transducer) includes a write element and a read element for writing and reading information stored on the medium 120 of the HDD 100, respectively. The medium 120 or multiple disk media may be attached to the spindle 124 using a disk clamp 128.

[0044] The HDD 100 also includes an arm 132 attached to the HGA 110, a carriage 134, a voice coil motor (VCM) including an armature 136 including a voice coil 140 attached to the carriage 134, and a stator 144 including a voice coil magnet (not visible). The armature 136 of the VCM is attached to the carriage 134 and is configured to move the arm 132 and the HGA 110 to access portions of the media 120, which are collectively mounted on a pivot 148 with an interposed pivot bearing assembly 152. In the case of an HDD with multiple disks, the carriage 134 may be referred to as an "E-block" or comb because the carriage is arranged to carry an array of linked arms, giving it the appearance of comb teeth.

[0045] The assembly including a head gimbal assembly (e.g., HGA110) including a flexure to which a head slider is coupled, an actuator arm (e.g., arm 132) and / or a load beam to which the flexure is coupled, and an actuator (e.g., VCM) to which the actuator arm is coupled may be collectively referred to as a head stack assembly (HSA). However, the HSA may include more or fewer components than those described. For example, the HSA may refer to an assembly that also includes an electrical interconnect component. Generally speaking, the HSA is an assembly configured to move a head slider to access portions of the medium 120 for read and write operations.

[0046] With further reference to FIG. 1 , electrical signals (e.g., current to the voice coil 140 of the VCM) including write signals to the head 110a and read signals from the head 110a are transmitted by a flexible cable assembly (FCA) 156 (or "flex cable"). The interconnect between the flexible cable 156 and the head 110a may include an arm electronics (AE) module 160, which may have an onboard preamplifier for the read signal and other read channel and write channel electronic components. The AE module 160 may be attached to the carriage 134, as shown. The flexible cable 156 may be coupled to an electrical connector block 164, which in some configurations provides electrical connectivity through an electrical feed-through provided by the HDD housing 168. The HDD housing 168 (or "housing base" or "substrate" or simply "base") together with the HDD cover provides a semi-sealed (or hermetically sealed, in some configurations) protective housing for the information storage components of the HDD 100.

[0047] Other electronic components, including a disk controller and servo electronics including a digital signal processor (DSP), provide electrical signals to the drive motor, the voice coil 140 of the VCM, and the head 110a of the HGA 110. The electrical signal provided to the drive motor causes the drive motor to rotate, thereby providing torque to the spindle 124, which in turn is transmitted to the medium 120 attached to the spindle 124. As a result, the medium 120 rotates in the direction 172. The rotating medium 120 forms an air cushion that acts as an air bearing on which the air bearing surface (ABS) of the slider 110b rides, so that the slider 110b flies above the surface of the medium 120 without making contact with the thin magnetic recording layer that records information. Similarly, in an HDD that utilizes a lighter-than-air gas (such as helium for non-limiting example), the rotating medium 120 forms an air cushion that acts as a gas or fluid bearing on which the slider 110b rides.

[0048] The electrical signal provided to the voice coil 140 of the VCM enables the head 110a of the HGA 110 to access the track 176 on which the information is recorded. As a result, the armature 136 of the VCM swings through the arc 180, which enables the head 110a of the HGA 110 to access the individual tracks on the medium 120. The information is stored in a plurality of radially nested tracks on the medium 120, which are arranged in sectors (such as sector 184) on the medium 120. Accordingly, each track is composed of a plurality of sectorized track portions (or "track sectors") such as sectorized track portion 188. Each sectorized track portion 188 may include recorded information and a data header containing error correction code information and a servo burst signal pattern, such as an ABCD-servo burst signal pattern (which is information identifying the track 176). When accessing track 176, a read element of head 110a of HGA 110 reads a servo burst signal pattern which provides a position error signal (PES) to the servo electronics, which in turn controls an electrical signal provided to the voice coil 140 of the VCM, thereby enabling head 110a to follow track 176. Upon finding track 176 and identifying a particular sectorized track portion 188, head 110a either reads information from track 176 or writes information to track 176 in accordance with instructions received by the disk controller from an external agent, such as a microprocessor of a computer system.

[0049] The electronic architecture of the HDD includes multiple electronic components for performing their respective HDD operating functions, such as a hard disk controller ("HDC"), an interface controller, an arm electronic module, a data channel, a motor driver, a servo processor, a buffer memory, etc. Two or more such components may be combined on a single integrated circuit board called a "system on a chip" ("SOC"). Several, if not all, of such electronic components are typically arranged on a printed circuit board that is coupled to the bottom side of the HDD, such as to the HDD housing 168.

[0050] Reference herein to hard disk drives, such as HDD 100 shown and described with reference to FIG. 1 , may include information storage devices sometimes referred to as “hybrid drives.” A hybrid drive generally refers to a storage device that has the functionality of a conventional HDD (see, e.g., HDD 100) combined with a solid-state storage device (SSD) that uses non-volatile memory, such as flash memory or other solid-state (e.g., integrated circuit) memory, which is electrically erasable and programmable. Because the operation, management, and control of different types of storage media are generally different, the solid-state portion of the hybrid drive may include its own corresponding controller functionality, which may be integrated into a single controller along with the HDD functionality. The hybrid drive may be constructed and configured to operate and utilize the solid-state portion in a variety of ways, such as, as a non-limiting example, using the solid-state memory as a cache memory, for storing frequently accessed data, for storing I / O intensive data, and the like. Additionally, the hybrid drive may be constructed and configured to be essentially two storage devices, i.e., a conventional HDD and an SSD, in a single housing, with one or more interfaces for host connection.

[0051] Solid State Drive Configuration

[0052] As discussed, embodiments may be used in the context of a data storage system that employs multiple data storage devices (DSDs), such as SSDs. Figure 1B is a block diagram illustrating an exemplary operating context in which embodiments of the present invention may be implemented. Figure 1B A general SSD architecture 150 is shown, wherein an SSD 152 is communicatively coupled to a host 154 via a host communication interface 156. Embodiments are not limited to Figure 1B Instead of the depicted configuration, embodiments may utilize Figure 1B For example, embodiments may be implemented to operate in other environments that rely on non-volatile memory storage components for writing and reading data.

[0053] Host 154 broadly represents any type of computing hardware, software, or firmware (or any combination of the foregoing) that makes data I / O requests or calls to one or more memory devices, etc. For example, host 154 can be an operating system executed on a computer, tablet, mobile phone, or any type of computing device that generally contains or interacts with memory, such as host 350 ( Figure 3 The host interface 156 that couples the host 154 to the SSD 152 may be, for example, an internal bus of the storage system or a communication cable or a wireless communication link, etc.

[0054] Figure 1BThe exemplary SSD 152 shown includes an interface 160, a controller 162 (eg, a controller having firmware logic therein), an addressing 164 functional block, a data buffer cache 166, and one or more non-volatile memory components 170a, 170b-170n.

[0055] Interface 160 is the point of interaction between components, namely SSD 152 and host 154 in this context, and is applicable at both the hardware and software levels. This allows the components to communicate with other components via input / output (IO) systems and associated protocols. Hardware interfaces are typically described by the mechanical, electrical, and logical signals at the interface and the protocols used to sequence them. Some non-limiting examples of common and standard interfaces include SCSI (Small Computer System Interface), SAS (Serial Attached SCSI), and SATA (Serial ATA).

[0056] SSD 152 includes controller 162 that incorporates electronics that bridge the nonvolatile memory components (e.g., NAND flash) to the host (such as nonvolatile memories 170a, 170b, 170n) to host 154. The controller is typically an embedded processor that executes firmware-level code and is an important factor in SSD performance.

[0057] The controller 162 interacts with the non-volatile memories 170a, 170b, 170n via an addressing 164 functional block. The addressing 164 functionality is used, for example, to manage the mapping between logical block addresses (LBAs) from the host 154 to corresponding physical block addresses on the SSD 152 (i.e., on the non-volatile memories 170a, 170b, 170n of the SSD 152). Because non-volatile memory pages and host sectors are different sizes, the SSD must build and maintain data structures that enable it to translate between the host that writes data to or reads data from a sector and the physical non-volatile memory page where the data is actually placed. This table structure or "mapping" may be built and maintained for sessions in the volatile memory 172 of the SSD, such as DRAM or some other local volatile memory component accessible to the controller 162 and addressing 164. Alternatively, the table structure may be maintained more persistently across sessions in non-volatile storage of an SSD, such as non-volatile storage 170a, 170b-170n.

[0058] In addition to the non-volatile memory 170a, 170b-170n, the addressing 164 also interacts with a data buffer cache 166. The data buffer cache 166 of the SSD 152 typically uses DRAM as a cache, similar to the cache in a hard disk drive. The data buffer cache 166 serves as a buffer or staging area for transferring data to and from the non-volatile memory components, and as a cache to speed up future requests for cached data. The data buffer cache 166 is typically implemented with volatile memory, so data stored therein is not permanently stored in the cache, that is, the data is not persistent.

[0059] Finally, the SSD 152 includes one or more non-volatile memory 170a, 170b-170n components. For non-limiting example, the non-volatile memory components 170a, 170b-170n may be implemented as flash memory (e.g., NAND or NOR flash) or other types of solid-state memory available now or in the future. The non-volatile memory 170a, 170b-170n components are the actual memory electronic components on which data is persistently stored. The non-volatile memory 170a, 170b-170n components of the SSD 152 may be considered to be an emulation of a hard disk in a hard disk drive (HDD) storage device.

[0060] In addition, references herein to data storage devices may encompass multi-media storage devices (or "multi-media devices," which may sometimes be referred to as "multi-layer devices" or "hybrid drives"). A multi-media storage device generally refers to a storage device having the functionality of a conventional HDD (see, e.g., HDD 100) combined with an SSD (see, e.g., SSD 150) that uses non-volatile memory, such as flash memory or other solid-state (e.g., integrated circuit) memory, which is electrically erasable and programmable. Because the operation, management, and control of different types of storage media are typically different, the solid-state portion of the hybrid drive may include its own corresponding controller functionality, which may be integrated into a single controller along with the HDD functionality. The multi-media storage device may be constructed and configured to operate and utilize the solid-state portion in a variety of ways, such as, by way of non-limiting example, using the solid-state memory as a cache memory, for storing frequently accessed data, for storing I / O intensive data, for storing metadata corresponding to payload data (e.g., for assisting in decoding the payload data), and the like. Additionally, the multi-media storage device may be constructed and configured as essentially two storage devices in a single housing, a conventional HDD and an SSD, with one or more interfaces for host connection.

[0061] Expansion and alternatives

[0062] In the foregoing description, embodiments of the present invention have been described with reference to a large number of specific details, which may vary according to different specific implementations. Therefore, various modifications and changes may be made to the embodiments without departing from the broader essence and scope of the embodiments. Therefore, the present invention and the applicant's sole and exclusive indicator of the present invention are a set of claims issued by this patent application in the specific form in which such claims are issued, including any subsequent corrections. Any definition of the terms contained in these claims explicitly set forth herein should determine the meaning of these terms used in the claims. Thus, limitations, elements, characteristics, features, advantages or attributes that are not explicitly cited in the claims should not limit the scope of this claim in any way. Therefore, this specification and the drawings are considered to be exemplary rather than restrictive.

[0063] Additionally, in this description, certain process steps may be shown in a particular order, and letters and alphanumeric labels may be used to identify certain steps. Unless expressly indicated in the specification, the embodiments are not necessarily limited to any particular order of performing such steps. In particular, these labels are used only to facilitate identification of the steps and are not intended to specify or require a particular order of performing such steps.

Claims

1. A method for identifying a problematic hard disk drive (HDD), the method comprising: retrieving a corresponding recovery log from each of a plurality of hard disk drives (HDDs); Determining a value of a full restore per hour (FRPH) metric for each respective read / write head of each respective HDD of the plurality of HDDs based on each respective restore log, comprising: summing the amount of time the head spends on all recovery steps performed by the head within a particular duration to perform the evaluation, including characterizing a full recovery as corresponding to a predetermined duration that the head spends on one or more recovery steps, determining an equal number of full restores within the specific duration based on the summed amounts of time, and determining the FRPH value of the head based on the determined number of full recoveries and the specific time duration; as well as In response to a particular head FRPH value reaching a predetermined FRPH threshold, an in-situ repair of the HDD in which the particular head operates is determined. 2 . The method according to claim 1 , wherein the plurality of HDDs constitute a single data storage system.

3. The method according to claim 2, further comprising: The FRPH threshold is determined based on an average value or a median value of the FRPHs of the plurality of HDDs constituting the single data storage system. The method according to claim 1 , wherein the plurality of HDDs constitute a plurality of data storage systems.

5. The method according to claim 4, further comprising: The FRPH threshold is determined based on an average value or a median value of the FRPHs of the plurality of HDDs constituting the plurality of data storage systems. The method according to claim 1 , wherein the specific duration is a configurable parameter.

7. The method of claim 1, wherein determining the value of FRPH further comprises identifying a problematic HDD as a HDD having one or more heads that exhibit a FRPH value equal to or greater than a fully recovered FRPH threshold over a period of 300 hours.

8. The method of claim 1 , wherein determining the in-situ repair of the HDD comprises: identifying the HDD as a problematic HDD based on the HDD having one or more heads that exhibit a FRPH value equal to or greater than the FRPH threshold value within the specific period of time; identifying a head having a highest FRPH value within the problematic HDD; as well as The head having the highest FRPH value is classified as ineligible for use in further input / output (IO) operations.

9. The method of claim 8, wherein determining the in-situ repair of the HDD comprises: The HDD is reformatted to its current storage capacity when no head exhibits a FRPH value equal to or greater than the FRPH threshold value within the specific period of time.

10. The method of claim 1, wherein determining the in-situ repair of the HDD comprises: identifying one or more problem areas of the storage medium that contribute to the FRPH value; as well as The one or more problem areas of the storage medium are marked as unavailable for further input / output (IO) operations.

11. A data storage system, comprising: multiple data storage devices; and A system controller circuit comprising a memory and one or more processors and implementing one or more sequences of instructions that, when executed by the one or more processors, cause the following operations to be performed: retrieving a corresponding recovery log from each of a plurality of hard disk drives (HDDs); Determining a value of a metric Full Recovery Per Hour (FRPH) metric for each respective read / write head of each respective HDD of the plurality of HDDs based on each respective recovery log, comprising: summing the amount of time the head spends on all recovery steps performed by the head within a particular duration, including characterizing a full recovery as corresponding to a predetermined duration that the head spends on one or more recovery steps, determining an equal number of full restores within the specific duration based on the summed amounts of time, and determining the FRPH value of the head based on the determined number of full recoveries and the specific time duration; as well as In response to a particular head FRPH value reaching a predetermined FRPH threshold, an in-situ repair of the HDD in which the particular head operates is determined.

12. The data storage system of claim 11, wherein the one or more instruction sequences cause the following operations to be performed: The FRPH threshold is determined based on an average value or a median value of FRPH of the plurality of HDDs constituting the data storage system.

13. The data storage system of claim 11, wherein the one or more instruction sequences cause the following operations to be performed: The FRPH threshold is determined based on an average value or a median value of the FRPHs of the plurality of HDDs constituting a plurality of data storage systems.

14. The data storage system of claim 11, wherein determining the in-situ repair of the HDD comprises: identifying the HDD as a problematic HDD based on the HDD having one or more heads that exhibit a FRPH value equal to or greater than the FRPH threshold value within the specific period of time; identifying a head having a highest FRPH value within the problematic HDD; as well as The head having the highest FRPH value is classified as ineligible for use in further input / output (IO) operations.

15. The data storage system of claim 14, wherein determining the in-situ repair of the HDD comprises: The HDD is reformatted to its current storage capacity when no head exhibits a FRPH value equal to or greater than the FRPH threshold value within the specific period of time.

16. The data storage system of claim 11, wherein determining the in-situ repair of the HDD comprises: identifying one or more problem areas of the storage medium that contribute to the FRPH value; as well as The one or more problem areas of the storage medium are marked as unavailable for further input / output (IO) operations.

17. A data storage system comprising: Multiple hard disk drives (HDD); means for retrieving a corresponding recovery log from each of the plurality of HDDs; Means for determining a value of a metric Full Recovery Per Hour (FRPH) metric for each respective read / write head of each respective HDD of the plurality of HDDs based on each respective recovery log, comprising: means for summing the amount of time spent by the head on all recovery steps performed by the head within a particular duration, including means for characterizing a full recovery as corresponding to a predetermined duration spent by the head on one or more recovery steps, means for determining an equal number of full restorations within said specific duration based on said summed amounts of time, and means for determining the FRPH value of the head based on the determined number of full recoveries and the specific duration; and Means for determining an in-situ repair of the HDD in which the particular head operates in response to the particular head FRPH value reaching a predetermined FRPH threshold.

18. The data storage system of claim 17, wherein the means for determining the in-situ repair of the HDD comprises: means for identifying the HDD as a problematic HDD based on the HDD having one or more heads that exhibit a FRPH value equal to or greater than the FRPH threshold value within the specific period of time; means for identifying a head having a highest FRPH value within said problematic HDD; and Means for classifying said head having said highest FRPH value as ineligible for use in further input / output (IO) operations.

19. The data storage system of claim 18, wherein the means for determining the in-situ repair of the HDD comprises: Means for reformatting the HDD to its current storage capacity if no head exhibits a FRPH value equal to or greater than the FRPH threshold value within the specific period of time.

20. The data storage system of claim 17, wherein the means for determining the in-situ repair of the HDD comprises: means for identifying one or more problem areas of a storage medium that contribute to said FRPH value; and Means for marking the one or more problem areas of the storage medium as unusable for further input / output (IO) operations.

21. A data storage system comprising: a plurality of solid-state data storage devices (SSDs); and A system controller circuit comprising a memory and one or more processors and implementing one or more sequences of instructions that, when executed by the one or more processors, cause the following operations to be performed: Determining a value of a wait metric for each respective SSD of the plurality of SSDs based on host input / output (IO) request conflicts includes: summing the amount of wait time spent by the SSD on conflicts within a particular duration, including characterizing the total wait as corresponding to a predetermined duration spent by the SSD to resolve conflicts, determining an equal number of full waits within the specific duration based on the summed wait time amounts, and determining the wait metric value of the SSD based on the determined number of full waits and the specific duration; as well as In response to a particular SSD wait metric value reaching a predetermined threshold, an in-situ repair of the SSD is determined.

Images