Multi-level erasure system with collaborative optimization

By adopting reconstruction modules and multi-level erasing technology in distributed data storage systems, the data storage device and parity configuration are coordinated to optimize data storage devices and parity configurations, and performance degradation and data loss problems during data reconstruction are solved, achieving rapid and effective data recovery and improve system reliability.

CN113282232BActive Publication Date: 2024-12-06SEAGATE TECH LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110162340.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-02-19
Filing Date
2021-02-05
Publication Date
2024-12-06
Estimated Expiration
2041-02-05

AI Technical Summary

Technical Problem

Distributed data storage systems face problems such as performance degradation and data loss when rebuilding data, especially due to the complexity and latency caused by the limited service life of data storage devices and data overhead.

Method used

The reconstruction module is used to generate and adjust the reconstruction strategy, and through the collaboration between the network controller and the device controller, the configuration of the data storage device and the allocation of parity data are optimized, and the data striping is dynamically adjusted using multi-level erase technology and prediction learning circuits. and parity volume configuration to reduce data reconstruction time and processing.

Benefits of technology

It realizes rapid and effective data reconstruction in the event of data storage device failure, reduces the risk of data loss and performance degradation, and improves data durability and overall system reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113282232B_ABST
    Figure CN113282232B_ABST
Patent Text Reader

Abstract

The present application discloses a multi-level erasure system with collaborative optimization. The data storage erasure system may have a host connected to a plurality of data storage devices via a network controller, wherein each of the plurality of data storage devices and the network controller are connected to a box controller and each of the plurality of data storage devices has a device controller. A reconstruction strategy may be generated using a reconstruction module connected to the plurality of data storage devices, the network controller, and the box controller. The reconstruction strategy may involve minimizing data reconstruction time when a failure of the plurality of data storage devices occurs by executing the reconstruction strategy in response to a detected or predicted failure in at least one of the plurality of data storage devices.
Need to check novelty before this filing date? Find Prior Art

Description

Summary of the invention

[0001] According to some embodiments, a data storage system has a host connected to a plurality of data storage devices via a network controller, wherein each of the plurality of data storage devices and the network controller are connected to a box controller and each of the plurality of data storage devices has a device controller. A reconstruction strategy is generated by a reconstruction module, which is connected to the plurality of data storage devices, the network controller, and the box controller. The reconstruction strategy may involve minimizing data reconstruction time when a failure occurs in the plurality of data storage devices by executing the reconstruction strategy in response to a detected failure in at least one of the plurality of data storage devices. The reconstruction module then changes the reconstruction strategy in response to the detected failure in the plurality of data storage devices.

[0002] Other embodiments of the data storage system connect a host to a plurality of data storage devices via a network controller, wherein each of the plurality of data storage devices and the network controller are connected to a box controller and each of the plurality of data storage devices has a device controller. A reconstruction strategy is generated by a reconstruction module, which is connected to the plurality of data storage devices, the network controller, and the box controller. The reconstruction strategy may involve minimizing data reconstruction time when a failure occurs in the plurality of data storage devices by executing the reconstruction strategy in response to a detected failure in at least one of the plurality of data storage devices. The reconstruction module then changes the reconstruction strategy in response to a failure in the plurality of data storage devices predicted by the reconstruction module.

[0003] In various embodiments, a system has a host connected to a plurality of data storage devices via a network controller, wherein each of the plurality of data storage devices and the network controller are connected to a box controller and each of the plurality of data storage devices has a device controller. A reconstruction strategy is generated using a reconstruction module, the reconstruction module being connected to the plurality of data storage devices, the network controller, and the box controller. The reconstruction strategy involves minimizing data reconstruction time when a failure occurs in the plurality of data storage devices. In response to a detected failure in at least one data storage device in the plurality of data storage devices, execution of the reconstruction strategy involves the device controller and at least one of the box controller and the network controller.

[0004] These and other features that may characterize various embodiments will be appreciated in view of the following detailed discussion and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 A functional block diagram representation of an example distributed data storage system is provided in which various embodiments may be practiced.

[0006] Figure 2 is a block diagram representation of a portion of an example data storage system arranged in accordance with some embodiments.

[0007] Figure 3 Depicted are block diagram representations of portions of an example data storage system configured in accordance with at least one embodiment.

[0008] Figure 4 Shown are block diagram representations of portions of an example data storage system employed in accordance with various embodiments.

[0009] Figure 5 Shown are block diagram representations of portions of an example data storage system arranged and operated in accordance with various embodiments.

[0010] Figure 6 Shown is a block diagram representation of an example reconstruction module that may be utilized in a data storage system in accordance with various embodiments.

[0011] Figure 7 Shown is a block diagram representation of portions of an example data storage system utilized in accordance with some embodiments.

[0012] Figure 8 Shown are block diagram representations of portions of an example data storage system arranged in accordance with various embodiments.

[0013] Fig. 9 Shown are block diagram representations of portions of an example data storage system arranged in accordance with various embodiments.

[0014] Fig.10 Depicted are block diagram representations of portions of an example data storage system arranged in accordance with various embodiments.

[0015] Fig.11 It can be used Figure 1-Figure 10 An example multi-level erase routine performed by various embodiments of the present invention. DETAILED DESCRIPTION

[0016] Various embodiments discussed herein generally relate to, but are not limited to, a distributed data storage system that intelligently employs multi-level erasure to provide optimized data durability and reduced cost.

[0017] The expansion of cloud data storage allows computing systems to become more streamlined and more efficient as data storage hardware is located offsite. The ability to store and access data from an offsite location allows computing systems to employ less onsite temporary and permanent data storage capabilities, thereby reducing the physical size and power consumption associated with operation. However, the evolution of large-scale distributed data storage, which can be characterized as cloud data storage, has experienced various challenges associated with providing secure, reliable, and fast data access.

[0018] Distributed data storage systems may store portions of data in multiple different data storage devices to provide enhanced data reliability, but failure of a data storage device may result in performance degradation when data is rebuilt within the system. Distribution of data across multiple data storage devices may further provide reduced data access latency by employing multiple data channels, but may suffer from the generation and maintenance of data overhead associated with distributed data storage. Such problems are exacerbated by the fact that many data storage devices, such as devices employing flash memory, have a limited lifespan corresponding to a limited number of data writes.

[0019] Thus, various embodiments optimize data storage by proactively and / or reactively configuring distributed data to provide fast and efficient data reconstruction operations in the event of a failure. Intelligent proactive configuration of data can reduce the volume and time associated with data reconstruction following data storage device errors and / or failures. Communication between multiple levels of a distributed data storage system can intelligently react to errors and / or failures to provide faster and more efficient reconstruction operations with improved data persistence.

[0020] Figure 1 1 is a block diagram representation of an example data storage system 100 in which various embodiments may be practiced. System 100 may be of any size and comprised of any size, number, and type of interconnected computing components. Figure 1 In a non-limiting configuration, multiple data storage devices 102 (DSD1-N) are respectively connected to at least one shared controller 104 residing in the distribution hardware 106. The shared controller 104 can direct data storage operations from any number of remote hosts to various data storage devices 102, or from various data storage devices 102 to any number of remote hosts.

[0021] Although not required or limiting, the system 100 may connect a first remote host 108 and a second remote host 110 to the distribution hardware 106. The remote hosts 108 / 110 may be physically separated while being connected to the shared controller 104 via a wired and / or wireless network 112. The shared controller 104 may treat some or all of the data storage devices 102 as a redundant array of independent disks / devices (RAID), as shown by the segmented lines with configurations such as levels 0, 4, 5, 6, etc., where data is striped and / or mirrored with or without parity data to provide reliable data storage.

[0022] The shared controller 104 can direct data flows to a single data storage device 102 and to a group of more than one data storage device 102, or from a single data storage device 102 and a group of more than one data storage device 102 to implement pending data access requests, such as data reads and data writes from the host 108 / 110. It is contemplated that the shared controller 104 can also direct data traffic in and between the data storage devices 102 as part of data storage maintenance or security. It is contemplated, but not required, that the various data storage devices 102 are physically present at different locations (such as different cities, states, countries, continents, or hemispheres) to increase system integrity in view of unit failures corresponding to geographical locations. The distribution hardware 106 and the remote hosts 108 / 110 can also be in different physical locations. Therefore, each aspect of the data storage system 100 can be located at different and unique physical locations, while being interconnected by the network 112.

[0023] Figure 2 A block diagram representation of portions of an example data storage system 120 arranged in accordance with some embodiments is depicted. The system 120 may have one or more network controllers 122 that operate to execute data access requests from one or more hosts 108 / 110. The network controller 122 may supplement or replace a shared controller 104 located in the distribution hardware 106. As a non-limiting example, the network controller 122 may be part of a server, router, switch, or node that interacts with the various hosts 108 / 110 on a front end, while the shared controller 104 is part of an enclosure or rack that houses multiple independent data storage devices 102. Another example may configure the network controller 122 and the shared controller 122 as similar types of network equipment (such as a server, router, or node) that have different operational tasks associated with writing data to and retrieving data from one or more connected data storage devices 102.

[0024] While the various data storage devices 102 incorporated into the distributed data storage system 120 may have different capabilities and configurations (such as capacity, error rates, and data access latency), it is contemplated that each data storage device 102 has a local controller 124 that performs at least data access and maintenance operations on a local memory 126. As such, the data storage devices 102 may employ rotating magnetic media, solid-state arrays, or a combination thereof to provide memory 126 that may be used to store data from one or more hosts 108 / 110. The physically separate positioning of the data storage devices 102 in the data storage system 120 allows user data from the various system hosts 108 / 110 to be distributed across multiple different devices 102 to prevent data storage errors and / or failures that are localized to a particular physical location, such as power outages or natural disasters.

[0025] Figure 3 A block diagram representation of portions of an example data storage system 130 configured in accordance with various embodiments is generally conveyed. The system 130 logically arranges a plurality of data storage devices 102 into columns 132 and rows 134, wherein each column 132 consists of different data storage addresses in one or more data storage devices 102 located at a single location, and each row consists of different data storage addresses in different data storage devices 102 located at different physical locations.

[0026] That is, the columns 132 are configured with data destinations in one or more data storage devices 102 that are physically located in a common location. For example, the columns 132 may be different physical block addresses (PBAs) in a single data storage device 102, or different physical block addresses (PBAs) in multiple data storage devices 102 located in different data packet shells at a single physical location. Figure 3 Each data storage device 102 of each column 132 shown in FIG. 1 is located at a common physical location, while each row 134 has different data storage devices 102 distributed across different data storage locations in different physical locations.

[0027] The logical configuration of logical block addresses (LBAs) provided into columns 132 and rows 134 by the various data storage devices 102 may be organized and executed using one or more controllers 136, which may be located locally (such as controller 124), located in distribution hardware 106 (such as controller 104), or located in a network (such as controller 122). The various controllers 136 of the data storage system 130 may operate independently, collectively, and sequentially with the local programmable hardware of the various data storage devices 102 comprising the system 130 to execute data access requests of the connected hosts 108 / 110 as well as data maintenance operations such as garbage collection, deduplication, and encryption.

[0028] Segmented wireframe 138 conveys how less than all of the LBAs in column 132 may be part of a common data storage device 102. Note, however, that all of the LBAs in column 132 may instead be part of multiple data storage devices 102. The ability to aggregate LBAs from one or more data storage devices 102 into a common logical column 132 allows controller 136 to freely form LBA 140 configurations to optimize performance regardless of the number, type, and capacity of available data storage devices 102.

[0029] It is noted that regardless of the number and physical location of the data storage devices providing the LBAs, the data capacities and data transfer speeds of the individual data storage devices 102 may be similar or dissimilar. Therefore, the controller 136 may organize and perform data storage operations, such as data writes, data reads, data updates, and data moves, on LBAs in data storage devices 102 having different data storage characteristics, such as write latency, read latency, and capacity.

[0030] The controller 136 may configure the data storage system 130 as a single RAID group, where user generated data (UD) is striped and / or mirrored with or without static or rotating parity data (PD1 and PD2) according to a pre-assigned RAID level (such as 0, 4, 5, or 6). Figure 3 As shown, controller 136 may configure data storage system 130 where each row 134 is a pre-assigned RAID group (G1 / G2 / G3 / G4) among which data is distributed according to a selected RAID level.

[0031] Despite the rotational parity provided by RAID levels 5 and 6, as opposed to the static parity provided by RAID level 4, the data storage system 130 may be susceptible to data loss and performance degradation in data reconstruction operations due to errors and / or failures encountered concurrently in multiple data storage devices 102. It is noted that utilizing multiple parity values ​​for blocks of user-generated data can provide system 130 resilience to two concurrent data storage device 102 failures, but can correspond to relatively long and complex data reconstruction operations. Further note that the use of uniform data stripes 142 composed of an equal number of user data portions and parity data portions allows for a patterned distribution of parity data such that not all parity is stored in a combined number of devices 102 and / or LBAs, which user data portions and parity data portions can be designated as 3+2, where each stripe 142 has 3 distributed user data portions and 2 distributed parity portions.

[0032] While maintaining a uniform data stripe 142 configuration can provide efficient data storage in a distributed model, such a uniform configuration can rarely be maintained over time because the LBAs of the data storage device 102 become prone to errors and / or cannot reliably store data. For example, failure of a block of LBAs in the solid-state data storage device 102 due to a large number of data writes will cause the uniform data stripe 142 configuration to be completely reconstructed into valid LBAs, or blocks of LBAs that are ignored are used for data storage, which over time reduces the actual capacity and efficiency of the system 130. In view of these issues, embodiments of the distributed data storage system involve utilizing irregular data stripe configurations.

[0033] Figure 4 A block diagram representation of portions of an example data storage system 150 configured and employed in accordance with various embodiments is shown. As shown, multiple data storage devices 102 are utilized at different physical locations to provide LBAs that are logically arranged into columns 154 and rows 156. Figure 3 Compared to uniform data stripes 140, system 150 concurrently stores data stripes having different configurations. In a non-limiting example, data stripe 158 (G1) has a 5+2 configuration stored in a single row 156 of LBAs, while data stripe 160 (G2) has a 4+2 configuration stored in multiple different rows 156.

[0034] The ability to change the number of user data partitions and the number of parity data blocks used for data striping provides flexibility and freedom to change data storage conditions over time to provide consistent performance and reliability. Figure 3Compared to uniform data stripes 140, the use of irregular data stripe configurations may result in performance degradation of the system 150 due to the greater amount of processing required to utilize the various LBAs 152. The data storage performance degradation associated with irregular data stripes may be particularly severe during data reconstruction operations when parity data is employed to reconstruct other portions of the data stripe. That is, due to the irregular data stripes, reconstruction of a portion of a column 154 may involve greater processing and reconstruction time than with uniform data stripes, in which a patterned amount of parity and user data will be reconstructed after a failure.

[0035] Where parity data and / or information is stored in multiple different levels of the distributed data storage system 140, multi-level erasure may be utilized to alleviate the complexity and processing time associated with rebuilding data in a distributed system 150 that employs irregular data striping. Figure 5 A block diagram representation of an example data storage system 170 employing a multi-level erase configuration according to various embodiments is shown. The system 170 may employ any number of data storage devices 102 that may be distributed across multiple physical locations (such as in different time zones) to store at least user data and associated parity data from one or more connected hosts 108 / 110.

[0036] Each data storage device 102 may be connected to a respective host 108 / 110 via one or more network components (such as servers, nodes, distribution hardware, routers, and switches) that service data access requests made by the various hosts 108 / 110 while providing robust data storage in the form of distributed user data portions and erase data. It is noted that the erase data may include any data and information related to the reconstruction of user data blocks and / or data structures of the data storage device 102. For example, the erase data may consist of parity data associated with user-generated data, redundant data mirrored from another LBA of the system 170, metadata associated with user-generated or non-user-generated data, or information related to the history, performance, organization, or structure of a data storage device and / or a portion of the data storage system 170.

[0037] By the organization of the various data storage components in the system 170, the data storage device 102 may be considered as device-level data storage, while multiple data storage devices 102 (such as data packets, racks, or single physical locations) may be considered as pods-level data storage, and the network components may be considered as network-level data storage. As non-limiting examples, device-level storage may consist of a single data storage device storing user-generated data and associated parity data for multiple different data stripes; pod-level storage may consist of a controller 172 corresponding to a shared controller 104 that directs the operation of multiple independent data storage devices that respectively store multiple different data stripes; and network-level storage may consist of a controller that directs multiple different groups of data storage devices.

[0038] As shown, logically organizing the data storage devices 102 into hierarchical levels allows for efficient erasure operations involving reconstruction of at least some of the data in at least one data storage device 102. Such hierarchical levels may correspond to different erasure resolutions that allow for rapid determination of parity data and other erasure information. For example, a network level may store erasure data having a first resolution corresponding to multiple different data storage devices in different data racks, a box level may store erasure data having a second resolution corresponding to multiple different data storage devices in a single rack, and a device level may store erasure data corresponding to a single data storage device.

[0039] A non-limiting practical example of using hierarchical erase levels stores erase data and / or information of different resolutions in each level, and in response to a data storage device error and / or failure, polls each network level, box level, and device level erase data in sequence to reconstruct the data stored in the error / failure data storage device. The configuration of different erase data resolutions allows the polling of erase data to be faster but more abstract at the network level, and slower but more thorough at the device level. As a result, if sufficient erase data is found at the network level or box level, device errors / failures can be handled more quickly than if each data storage device at the device level were exhaustively polled for erase data when an error / failure occurs.

[0040] While the use of different erase data levels and resolutions may result in higher data reconstruction efficiency than erase data stored at a single resolution, the durability of data and the performance of the hierarchical multi-level erase system 170 may be optimized by increasing communication and coordination between the levels. Accordingly, various embodiments are directed to improving the use of the multi-level erase system 170 to reduce the time and processing corresponding to the reconstruction / reconstruction of data and / or data structures following a data storage device error or failure.

[0041] Figure 6A block diagram representation of an example reconstruction module 180 that may be implemented in hardware and utilized in a data storage system in accordance with various embodiments is depicted. The reconstruction module 180 may be constructed as one or more circuits and programmable hardware residing in a single data storage device 102, distribution hardware 106, network controller 122, or host 108. It is contemplated that multiple reconstruction modules 180 may be part of a data storage system and utilized redundantly, sequentially, or independently in similar or dissimilar capabilities and configurations.

[0042] The reconstruction module 180 may have a local controller 182 (such as a programmable circuit or processor) that directs input of at least current data storage conditions, current data storage performance, data error rates, and data stripe configurations to generate a reconstruction strategy that provides proactive and reactive measures to be taken to reduce the occurrence of data storage device failures and reduce the time associated with repairing data and data structures in response to encountered data storage device errors / failures. The reconstruction module 180 may utilize the local controller 182 alone, or in conjunction with other controllers 104 / 122 / 136 of the data storage system 180, to generate, modify, and otherwise maintain the reconstruction strategy in a form that best optimizes data reconstruction operations.

[0043] The reconstruction module 180 may have a prediction circuit 184 that converts input system / data information into at least a reconstruction time associated with a hypothetical failure at one or more locations in the system. For example, the prediction circuit 184 may simulate a data failure in a single data storage device and a data failure in multiple data storage devices in one or more separate locations to predict how long such failures will take to rebuild the corrupted data. The learning circuit 186 of the reconstruction module 180 may supplement the prediction circuit 184 to improve the accuracy of the predicted data reconstruction time. That is, the learning circuit 186 may record past data reconstruction operations to data storage devices, data stripes, data racks, and physical locations to learn predictable aspects of data reconstruction for various aspects of the distributed data storage system. In this way, the learning circuit 186 may be rear-facing in response to past system events, while the prediction circuit 184 is forward-facing by predicting future system events.

[0044] The ability to understand past system events and accurately predict future data events can ensure that the reconstruction strategy can mitigate the performance degradation of data storage device errors / failures. By incorporating the reconstruction circuit 188 into the reconstruction module 180, the hardware can also obtain current and historical data storage system information and conditions to evaluate different proactive and reactive reconstruction scenarios and their impact on reconstruction time and stored data vulnerability. The reconstruction circuit 188 can predict a variety of different proactive data movements, stripe configurations, and reconstruction processing options alone or in combination with the prediction circuit 184 to ensure that the reconstruction strategy prescribes the best measures to minimize the reconstruction of data and / or data structures.

[0045] The reconstruction circuitry 188 can evaluate the impact of various reactive actions, communications, and processing that can be performed by device-level, box-level, and network-level components on the reconstruction time. Thus, the reconstruction module 180 can predict, learn, and evaluate how different proactive and reactive actions will affect the reconstruction time based on current system conditions. Figure 7-10 Non-limiting examples of implementations of the reconstruction strategies are conveyed respectively. Note that corresponding implementations are not required and the reconstruction strategy may involve executing multiple different proactive and / or reactive activities concurrently, sequentially or independently.

[0046] Figure 7 Depicted is a block diagram representation of portions of an example data storage system 200 utilized in accordance with some embodiments to implement a rebuild strategy generated from a rebuild module 180. In response to an error / failure 202 in a data storage device 102, a device-level controller 136 executes a rebuild strategy by polling at least one downstream, box-level, and / or network-level controller 104 / 122 to determine which LBAs in the failure 202 contain valid data. For example, the rebuild strategy may react to a detected failure 202 in some or all of the LBAs of the data storage device 102 by querying an upper-level controller to inquire which LBAs in the failed device 102 contain current valid data (as represented by arrows 204 and 206).

[0047] It is contemplated that the polled superior controller 104 / 122 does not simply return the location of valid data, such as the status of each of the LBAs of the failed portion of the data. For example, the polled superior controller 104 / 122 may return if the LBA is empty, a non-current version of the data, an error / failure, parity, a mirror, or a current version of the data. By utilizing the superior controller(s) 104 / 122 to determine which LBA of the failed device 102 contains valid data rather than the local device controller, it allows for ensuring that the failed LBA is rebuilt to effectively reconstruct only valid data. Such avoidance of rebuilding invalid data and error-prone LBAs eliminates unnecessary processing and reduces rebuild time.

[0048] While it is possible that each data storage device 102 can maintain a table of valid data, invalid data, and error-prone LBAs, the maintenance of such a table would greatly increase the processing operations performed by the local controller. In addition to the degraded performance associated with the increased volume processed by the local controller, the storage and frequent updating of information identifying the state of each LBA in the data storage device 102 would quickly reach a limited threshold for writing data to the solid-state memory. Therefore, maintaining the state of the LBA by one or more upper-level controllers 104 / 122 can be more efficiently processed and utilize various storage locations for the state table compared to the local controller 124.

[0049] Figure 8 A block diagram representation of portions of an example data storage system 210 utilized in accordance with various embodiments to implement a reconstruction strategy generated from a reconstruction module 180 is shown. In response to a failure 212 of one or more data storage devices 102 being estimated to take a relatively long time to rebuild, an upper level controller 104 / 122 (box level and / or network level) may direct a non-failed device 214 to temporarily increase data durability of at least one data stripe. The increase in data durability may involve changing a data stripe to include one or more volumes of additional parity, such as writing more parity in different devices 102 and locations.

[0050] Some embodiments increase the durability of data stripes in non-failed devices 214 to compensate for data stripes in failed devices 212. For example, a data stripe having two volumes of parity data associated with eight volumes of user-generated data may prompt compensation by writing parity data of additional volumes to other non-failed data stripes. Such compensation is particularly effective in multi-level erasure systems where data stripes have different numbers of parity volumes, such as if a first-level stripe has ten data volumes and one parity volume, and a second-level stripe has eight data volumes and two parity volumes, then compensation may be performed by increasing the parity volumes to three or more before failing over to the second level.

[0051] The ability to temporarily or permanently increase data durability in response to a failure 212, or proactively anticipate a predicted failure in a reconstruction strategy, allows more parity data volumes and locations to be used for data stripe reconstruction operations. In the event of a catastrophic failure of multiple data storage devices 102, the increase in the number of parity data volumes can significantly reduce the time associated with rebuilding multiple data stripes across multiple data storage devices 102. It is contemplated that the added parity data volumes can be temporarily written to any location in the distributed data storage system 210 (such as other data storage devices, non-failed portions of the data storage device that caused the failure 212, a network cache, or a remote buffer).

[0052] Fig. 9 A block diagram representation of portions of an example data storage system 220 utilized in accordance with various embodiments to implement a reconstruction strategy generated from a reconstruction module 180 is depicted. Maintenance of a volume of parity data allows data stripes to be reconstructed despite the loss of multiple data storage devices 102. The local device controller 124 may perform one or more reconstruction operations to repair at least one data stripe from the volume of parity data. However, such reconstruction operations may be processing intensive and relatively slow.

[0053] Through the implementation of the reconstruction strategy, the local controller 124 can employ one or more upper level controllers 104 / 122 to assist in the reconstruction of one or more data stripes, as represented by arrows 222 and 224. For example, the local controller 124 can initially distribute half of the overall data reconstruction calculations to one or more upper level controllers 104 / 122. In another example, the local controller 124 can perform all reconstruction calculations and processing within a threshold amount of time specified by the reconstruction strategy before passing some or all of the reconstruction processing / computation to one or more upper level controllers 104 / 122. The ability to proactively and reactively employ other processing components of the distributed data storage system 220 allows the local controller 124 to receive assistance with reconstruction operations to improve reconstruction time without degrading the performance of the processing components.

[0054] It is contemplated that a rebuild strategy and the local controller 124 executing the strategy may dictate the rebuilding of certain data, data stripes, or data types to a particular controller 104 / 122. Such assignment of rebuilds to a particular controller 104 / 122 may be consistent with a higher level of data security, a data stripe configuration that will take longer to rebuild with a local controller 124, or the processing capabilities of the assigned controller relative to the local controller. Some embodiments of the implemented rebuild strategy concurrently use multiple different controllers 124 / 104 / 122 in processing and computing data for rebuilding some or all of the data stripes.

[0055] The ability to employ controller(s) to assist in reconstructing fault data is not limited to the superordinate controller 104 / 122 . Fig.10 A block diagram representation of portions of an example data storage system 230 configured in accordance with some embodiments is depicted that is utilized to implement a reconstruction strategy generated from a reconstruction module 180. The system 230 employs one or more local controllers of non-failed data storage devices 102 to assist in performing reconstruction operations, as represented by arrows 232 and 234. Using other local controllers 124 in place of the upper level controller 104 / 122 may improve the performance of erasure coding and reconstruction using hardware and reduce workload compared to the upper level controller 104 / 122. For example, the local controller 124 may have a data reconstruction application specific integrated circuit that may provide efficient data reconstruction from one or more parity volumes.

[0056] Although not limiting or required, using other controllers 104 / 122 / 124 to perform a rebuild operation may involve determining what data of the failed LBAs of one or more data storage devices 102. That is, instead of rebuilding each LBA of the failed device 102, one or more system controllers 104 / 122 / 124 may calculate the contents of the failed LBAs and rebuild the data that was actually lost due to the failure. The determination of the contents of the failed LBAs may be further used to prioritize or schedule the reconstruction of the data, such as rebuilding severely degraded data stripes, sensitive data, or only the parity data portion of a data stripe first or last.

[0057] Fig.11 Shows that you can use Figure 1-Figure 10An example multi-level erase routine 250 is performed by various embodiments of the present invention. Initially, a distributed data storage system is arranged and interconnected so that at least one remote host can direct data access operations to or from at least one data storage device. It is contemplated that multiple data storage devices are arranged for data storage requested by a host without the host knowing or having the ability to control what data storage device will service the request. The distributed data storage system can position one or more controllers at different hierarchical levels connected to the reconstruction module, such as a network level, a box level, and a device level.

[0058] The receipt of new host data prompts the calculation and storage of data stripes across multiple data storage devices of the system in step 252. In a single-level erase system, a single data stripe would be created across multiple independent data storage devices having a predetermined number of user data volumes and a predetermined number of corresponding parity data volumes. With a multi-level erase system, multiple levels of at least data stripe parity volumes are created and stored at the corresponding system level (network / box / device). Thus, steps 254, 256, and 258 proceed to generate erase data, which is at least host-provided parity volumes, user-generated data for network-level controllers, box-level controllers, and device-level controllers.

[0059] The erase data for the respective system levels may be configured in different hierarchical arrangements. For example, the highest level may have a single parity volume, while the box level has two parity volumes, and the device level has three parity volumes for each data stripe stored in the data storage devices of the box. That is, the device level erase data may reside in the data storage device, and the box level erase data may relate to the parity data for each data stripe stored in each of the data storage devices controlled by a box level controller (such as a rack or server controller located at the same physical location as the respective connected data storage devices).

[0060] Generating different amounts of parity data in various levels of the distributed data storage system allows for efficient hierarchical polling of parity data, which can reduce workload and latency in local devices or boxes to rebuild data after a failure / error has damaged one or more data stripes. Although not required, step 260 can utilize parity data from one or more levels to reconstruct some or all of the user-generated data that was damaged in the encountered failure / error. Routine 250 can loop through steps 252-260 to receive new data, generate parity data, and use the parity data to reconstruct damaged data according to a reconstruction strategy.

[0061] However, it is contemplated that the reconstruction module may continuously or sporadically evaluate data storage conditions and performance in one or more data storage devices to determine whether a change to an existing reconstruction strategy can improve the risk of data storage device failure and / or reduce the time to complete data reconstruction after encountering an error / failure in decision 262. The reconstruction strategy change will benefit from the determination result of decision 262 prompting step 264 to generate a new reconstruction strategy, which is then put into place by at least one level of the distributed data storage system in step 266.

[0062] In some embodiments, decision 262 may determine that one or more proactive actions may be performed to reduce the risk of data storage device failure / error. For example, the reconstruction strategy may be changed to divide the user-generated data into more, fewer, larger, or smaller partitions that span more or fewer data storage devices to reduce the likelihood of the devices encountering errors. It is contemplated that the data stripes in the altered reconstruction strategy may maintain the stripe configuration, but may be moved to different groups of data storage devices to avoid error-prone devices. Such movement of data stripes may separate parity data from corresponding user-generated data.

[0063] The reconstruction strategy may alternatively be changed to change the parity configuration of one or more levels (network / box) to improve the persistence of the data stripes. Thus, decision 262 may evaluate the current persistence of the stored data for failures / errors, and modify the current data stripe configuration and / or reconstruction strategy to mitigate the risk of permanent data loss and long reconstruction times. It is noted that persistence may be characterized as resilience to errors / failures of the data storage device. Thus, as the number of parity data volumes that can be used to rebuild failed data increases, data persistence increases. Persistence may also be improved by moving parity data volumes to data storage locations that are less prone to errors and failures that would require reconstruction of the parity data.

[0064] The ability to proactively change data striping and parity configurations at one or more levels of the system in response to current and / or predicted system conditions allows for accurate and efficient management of the system without degrading data storage performance. By generating rebuild strategies that perform proactive and / or reactive actions to detected system conditions, data rebuild time can be minimized and data storage failures can be avoided and / or mitigated.

[0065] Typically, when a lower level block erase system loses a drive, the system does not know which of the lost blocks hold valid data and which do not. If the system knew this, then the rebuild operation could be improved by only rebuilding blocks that hold valid data, since rebuilding blocks that hold invalid data is unnecessary wasted work. Therefore, various embodiments introduce an API by which a lower level block erase system can query a higher level object erase system to ask which blocks are valid.

[0066] It is contemplated that when the upper-level erase system needs to recover from a catastrophic failure of the entire lower-level block erase system, the system will need to transfer a large amount of data across the network, which is necessary but may be slow. If the operation is slow, data persistence will be compromised because additional failures may occur during the slow reconstruction. Therefore, some embodiments introduce an API through which the upper-level can request other lower-level erase systems to temporarily increase their internal persistence, thereby restoring the desired persistence level more quickly. Once the upper-level reconstruction is complete, the system can signal the lower-level to remove the temporarily increased additional persistence. For example, suppose we have 10+1 at the upper-level and 8+2 at the lower-level. If we lose the entire lower-level system, our 10+1 at the upper-level can recover the lost data, but this will be slow. Therefore, it can temporarily request the relevant lower-level system to temporarily increase to 8+3 to compensate for the lost persistence at the external level.

[0067] When the lower level block erase system loses data irretrievably (for example, 8+2 and 3 drives fail at the same time), the system can request the upper level to recover the lost data. By adding an API for this purpose, the lower level can request the upper level to calculate the data for a specific block through the API, so that the operation can be optimized.

[0068] Embodiments can optimize the performance and cost of erasure coding within a low-level erase block system. To further improve performance and reduce cost, an API can be added so that a superior object erase system can request a subordinate block erase system to calculate erases for it, thereby allowing all levels in a multi-level erase system to use optimized erase units. Thus, a multi-level erase system is built on top of block devices that interact with those block devices only via a very narrow and constrained block interface, which allows a richer interface than a standard block interface, thereby optimizing the operation of a multi-block erase system.

[0069] Further examples

[0070] Example 1. A method comprising:

[0071] connecting the host to a plurality of data storage devices via a network controller, each of the plurality of data storage devices and the network controller being connected to a box controller, each of the plurality of data storage devices comprising a device controller;

[0072] A reconstruction strategy is generated using a reconstruction module connected to the plurality of data storage devices, the network controller, and the enclosure controller, the reconstruction strategy involving minimizing data reconstruction time when a failure of the plurality of data storage devices occurs.

[0073] executing the rebuild strategy in response to a detected failure in at least one of the plurality of data storage devices; and

[0074] The reconstruction strategy is altered using the reconstruction module in response to a detected change in a data storage condition.

[0075] Example 2. The method of Example 1, wherein the reconstruction module includes a controller that directs the operation of the reconstruction circuit.

[0076] Example 3. The method as described in Example 1 is characterized in that the reconstruction module maintains a log of data storage operations for the multiple data storage devices, and the log is used to generate the reconstruction strategy.

[0077] Example 4. The method of Example 1, wherein the changed reconstruction strategy improves data durability of at least one data stripe stored in the plurality of data storage devices.

[0078] Example 5. The method of Example 4, wherein the data durability is improved by writing at least one additional parity volume for the data stripe.

[0079] Example 6. The method of Example 5, wherein the additional parity volume is stored in a data storage device among the plurality of data storage devices that does not include a parity volume for the data stripe.

[0080] Example 7. The method of Example 1, wherein the device controller first executes a first part of the rebuild strategy until a predetermined threshold of the rebuild strategy is met, and then the box controller executes the remaining part of the rebuild strategy.

[0081] Example 8. The method of Example 1, wherein the reconstruction strategy is executed to reconstruct at least one user-generated data volume from the data stripes lost in the detected failure.

[0082] Example 9. A method comprising:

[0083] connecting the host to a plurality of data storage devices via a network controller, each of the plurality of data storage devices and the network controller being connected to a box controller, each of the plurality of data storage devices comprising a device controller;

[0084] A reconstruction strategy is generated using a reconstruction module connected to the plurality of data storage devices, the network controller, and the enclosure controller, the reconstruction strategy involving minimizing data reconstruction time when a failure of the plurality of data storage devices occurs.

[0085] executing the rebuild strategy in response to a first detected failure in at least one of the plurality of data storage devices; and

[0086] The reconstruction strategy is altered using the reconstruction module in response to a change in a data storage condition predicted by the reconstruction module.

[0087] Example 10. The method of Example 9, wherein the reconstruction module includes a prediction circuit that predicts at least one change in the data storage condition based on current data storage conditions detected in the plurality of data storage devices.

[0088] Example 11. The method of Example 9, wherein the reconstruction module includes a learning circuit that associates past recorded data storage conditions with future data storage conditions in the plurality of data storage devices.

[0089] Example 12. The method of Example 9, wherein the changed reconstruction strategy changes the data stripe by adding at least one parity volume in response to the predicted change in the data storage condition.

[0090] Example 13. The method of Example 9, wherein the changed reconstruction strategy temporarily increases the number of the parity volumes stored at the network level by the network controller in response to the predicted change in the data storage condition.

[0091] Example 14. The method of Example 9, wherein the changed rebuild strategy temporarily increases the number of parity volumes stored at the enclosure level by the enclosure controller in response to the predicted change in the data storage condition.

[0092] Example 15. The method of Example 9, wherein the changed rebuild strategy temporarily increases the number of the parity volumes stored at the device level by the device controller in response to the predicted change in the data storage condition.

[0093] Example 16. The method of Example 9, wherein the changed reconstruction strategy adds at least one of the parity volumes to the box level or the network level to balance the loss of the parity volume at the device level associated with the first detected failure.

[0094] Example 17. The method of Example 10, wherein the changed rebuild strategy employs the box controller having a controller to rebuild data lost due to the second detected failure.

[0095] Example 18. The method of Example 10, wherein the changed reconstruction strategy employs the network controller and the box controller to rebuild the data lost due to the second failure.

[0096] Example 19. The method of Example 10, wherein the changed reconstruction strategy uses multiple device controllers of independent data storage devices among the multiple data storage devices to rebuild the data lost due to the second failure.

[0097] Example 20. A method comprising:

[0098] connecting the host to a plurality of data storage devices via a network controller, each of the plurality of data storage devices and the network controller being connected to a box controller, each of the plurality of data storage devices comprising a device controller;

[0099] generating a reconstruction strategy using a reconstruction module connected to the plurality of data storage devices, the network controller, and the enclosure controller, the reconstruction strategy involving minimizing data reconstruction time when a failure occurs in the plurality of data storage devices; and

[0100] The rebuild strategy is executed using the device controller and at least one of the enclosure controller and the network controller in response to a detected failure in at least one of the plurality of data storage devices.

Claims

1. A method for data storage, comprising: connecting the host to a plurality of data storage devices via a network controller, each of the plurality of data storage devices and the network controller being connected to a box controller, each of the plurality of data storage devices comprising a device controller; generating a reconstruction strategy using a reconstruction module connected to the plurality of data storage devices, the network controller, and the enclosure controller, the reconstruction strategy involving minimizing data reconstruction time when a failure occurs in the plurality of data storage devices; executing the rebuild strategy in response to a detected failure in at least one of the plurality of data storage devices; executing, by the device controller, a first portion of the rebuild strategy until a predetermined threshold of the rebuild strategy is met; executing, with the box controller, the remainder of the rebuild strategy; and The reconstruction strategy is altered using the reconstruction module in response to a detected change in a data storage condition.

2. The method according to claim 1, characterized in that The reconstruction module includes a controller that directs operation of a reconstruction circuit.

3. The method according to claim 1, characterized in that The reconstruction module maintains a log of data storage operations for the plurality of data storage devices, the log being used to generate the reconstruction strategy.

4. The method according to claim 1, characterized in that The changed reconstruction strategy improves data durability of at least one data stripe stored in the plurality of data storage devices.

5. The method according to claim 4, characterized in that The data durability is improved by writing at least one additional parity volume for the data stripe.

6. The method according to claim 5, characterized in that The additional parity volume is stored in a data storage device that does not include a parity volume for the data stripe among the plurality of data storage devices.

7. The method according to claim 1, characterized in that The predetermined threshold of the rebuild strategy is an amount of time.

8. The method according to claim 1, characterized in that The reconstruction strategy is executed to reconstruct at least one user-generated data volume from the data stripes lost in the detected failure.

9. A method for data storage, comprising: connecting the host to a plurality of data storage devices via a network controller, each of the plurality of data storage devices and the network controller being connected to a box controller, each of the plurality of data storage devices comprising a device controller; generating a reconstruction strategy using a reconstruction module connected to the plurality of data storage devices, the network controller, and the enclosure controller, the reconstruction strategy involving minimizing data reconstruction time when a failure occurs in the plurality of data storage devices; executing the rebuild strategy in response to a first detected failure in at least one of the plurality of data storage devices; as well as In response to a predicted change in a data storage condition predicted by the reconstruction module, the reconstruction strategy is changed using the reconstruction module to change data striping by adding at least one parity volume.

10. The method according to claim 9, characterized in that The reconstruction module includes prediction circuitry that predicts at least one change in the data storage conditions based on detected current data storage conditions in the plurality of data storage devices.

11. The method according to claim 9, characterized in that The reconstruction module includes a learning circuit that correlates past recorded data storage conditions with future data storage conditions in the plurality of data storage devices.

12. The method according to claim 9, characterized in that The altered reconstruction strategy changes data striping by adding at least two parity volumes in response to the predicted change in the data storage condition.

13. The method according to claim 9, characterized in that The altered reconstruction strategy temporarily increases the number of parity volumes stored at a network level by the network controller in response to the predicted change in the data storage condition.

14. The method according to claim 9, characterized in that The altered rebuild strategy temporarily increases the number of parity volumes stored at the enclosure level by the enclosure controller in response to the predicted change in the data storage condition.

15. The method according to claim 9, characterized in that The altered rebuild strategy temporarily increases the number of parity volumes stored at a device level by the device controller in response to the predicted change in the data storage condition.

16. The method according to claim 9, characterized in that The altered reconstruction strategy adds at least one parity volume to a box level or a network level to balance the loss of parity volumes at the device level associated with the first detected failure.

17. The method according to claim 10, characterized in that The altered rebuild strategy employs the box controller with the controller to rebuild data lost due to the second detected failure.

18. The method according to claim 10, characterized in that The altered reconstruction strategy employs the network controller and the box controller to reconstruct the data lost due to the second failure.

19. The method according to claim 10, characterized in that The altered reconstruction strategy employs a plurality of device controllers of independent data storage devices of the plurality of data storage devices to reconstruct data lost due to the second failure.

20. A method for data storage, comprising: connecting the host to a plurality of data storage devices via a network controller, each of the plurality of data storage devices and the network controller being connected to a box controller, each of the plurality of data storage devices comprising a device controller; generating a reconstruction strategy using a reconstruction module connected to the plurality of data storage devices, the network controller, and the enclosure controller, the reconstruction strategy involving minimizing data reconstruction time when a failure occurs in the plurality of data storage devices; as well as In response to a detected failure in at least one of the multiple data storage devices, the reconstruction strategy is executed using the device controller and at least one of the box controller and the network controller, a first portion of the reconstruction strategy is executed using the device controller until a predetermined threshold of the reconstruction strategy is met, and then the remaining portion of the reconstruction strategy is executed using the box controller.