Systems and methods for repair redundancy control for large erasure coded data storage

Abstract

Systems and methods which implement storage system data repair control techniques, such as may provide a feedback control structure for regulating source object redundancy and/or repair bandwidth in the storage system. Embodiments control a source object redundancy level to be used in a storage system by analyzing source objects represented in a repair queue to determine repair rate metrics for the source objects and determining a source object redundancy level based on the repair rate metrics. For example, embodiments may cause more redundant fragments for each source object to be generated and stored during repair where the repair rate metrics indicate an increase in storage node failure rate. Additionally, embodiments may determine a per storage object repair rate (e.g., a repair rate preference for each of a plurality of source objects) and select a particular repair rate (e.g., a maximum repair rate) for use by a repair policy.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]This application claims the benefit of U.S. Provisional Patent Application No. 62 / 211,303, entitled, “SYSTEMS AND METHODS FOR REPAIR RATE CONTROL FOR LARGE ERASURE CODED DATA STORAGE”, filed on Aug. 28, 2015, which is expressly incorporated by reference herein in its entiretyDESCRIPTION OF THE RELATED ART[0002]The creation, management, storage, and retrieval of electronic data has become nearly ubiquitous in the day-to-day world. Such electronic data may comprise various forms of information, such as raw data (e.g., data collected from sensors, monitoring devices, control systems, etc.), processed data (e.g., metrics or other results generated from raw data, data aggregations, filtered data, etc.), produced content (e.g., program code, documents, photographs, video, audio, etc.), and / or the like. Such data may be generated by various automated systems (e.g., network monitors, vehicle on-board computer systems, automated control systems, e...

AI Technical Summary

Benefits of technology

[0017]A non-transitory computer-readable medium comprising codes for controlling a source object repair rate in a storage system is provided according to yet further embodiments herein. The codes of embodiments cause a computer to analyze a plurality of source objects in a repair queue to determine at least one repair rate metric for the source objects of the plurality of source objects. The codes of embodiments furth

Problems solved by technology

However, the individual storage nodes are somewhat unreliable in that they can intermittently fail, in which case the data stored on them is temporarily unavailable, or permanently fail, in which case the data stored on them is permanently lost (e.g., as represented by the failure of storage node 130-2 in FIG. 1C).

This can result in inefficiencies with respect to the use of resources, such as communication bandwidth, computing resources, etc.

Because a large amount of data is stored on each node (typically many terabytes), and all data comprising fragments stored on a node typically needs to be replaced when the node permanently fails, the repair process reads and writes a large amount of data after a node permanently fails.

This reading and writing of large amounts of data generally results in the consumption of large amounts of ba

Images