Systems and methods for verification of code resiliency for data storage

Abstract

Systems and methods which implement forward checking of data integrity are disclosed. A storage system of embodiments may, for example, comprise data integrity forward checking logic which is operable to perform forward checking of data integrity in real-time or near real-time to check that a number of node failures can be tolerated without loss of data. Embodiments may be utilized to provide assurance that a number of fragments needed for source data recovery will be available for the source objects most susceptible to failure when a certain number of additional fragments are lost, such as due to storage node failures.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]This application claims the benefit of U.S. Provisional Patent Application No. 62 / 211,243, entitled, “SYSTEMS AND METHODS FOR VERIFICATION OF CODE RESILIENCY FOR DATA STORAGE”, filed on Aug. 28, 2015, which is expressly incorporated by reference herein in its entirety.DESCRIPTION OF THE RELATED ART[0002]The creation, management, storage, and retrieval of electronic data has become nearly ubiquitous in the day-to-day world. Such electronic data may comprise various forms of information, such as raw data (e.g., data collected from sensors, monitoring devices, control systems, etc.), processed data (e.g., metrics or other results generated from raw data, data aggregations, filtered data, etc.), produced content (e.g., program code, documents, photographs, video, audio, etc.), and / or the like. Such data may be generated by various automated systems (e.g., network monitors, vehicle on-board computer systems, automated control systems, etc.), b...

AI Technical Summary

Benefits of technology

The patent describes a method and apparatus for analyzing source objects stored by a storage system as a plurality of erasure coded fragments. The method involves obtaining a prospective number of fragment losses for verifying source data integrity with respect to the storage system, and analyzing a set of erasure coded fragments commonly available for the source objects to determine if the storage system is susceptible to source data loss due to a loss of one of the fragments. The technical effect of this patent is to provide a reliable method for ensuring data integrity in a storage system by analyzing the set of erasure coded fragments commonly available for source objects and determining if the system is susceptible to source data loss due to a loss of one of the fragments.

Problems solved by technology

However, the individual storage nodes are somewhat unreliable in that they can intermittently fail, in which case the data stored on them is temporarily unavailable, or permanently fail, in which case the data stored on them is permanently lost (e.g., as represented by the failure of storage node 130-2 in FIG. 1C).

This can result in inefficiencies with respect to the use of resources, such as communication bandwidth, computing resources, etc.

Because a large amount of data is stored on each node (typically many terabytes), and all data comprising fragments stored on a node typically needs to be replaced when the node permanently fails, the repair process reads and writes a large amount of data after a node permanently fails.

This reading and writing of large amounts of data generally results in the consumption of large amounts of bandwidth in the storage system by the repair policy and, when coupled with a reactive repair policy, can result in disruptive spikes in storage system bandwidth utilization (e.g., delaying or preventing primary data access services by the storage system).

That is, the repair policy implemented with respect to a small erasure code configuration may act somewhat erratically whereby it utilizes a large amount of bandwidth for periods of time when attempting to repair data immediately upon determining fragments are unavailable due to a permanently failed node.

Furthermore, the repair process may not be able to repair fragments fast enough before further node failures lead to permanent loss of object data if not enough bandwidth is allocated to the repair process.

Images