Highly available transaction logs for storing multi-tenant data sets on shared hybrid storage pools

Abstract

Creating and using highly available transaction logs in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server receives a transaction entry at a first storage pool of the distributed storage system. The storage controller server further looks up a transaction log to store the transaction entry, where the transaction log is associated with a second storage pool. The storage controller server routes the transaction entry to the second storage pool, wherein the second storage pool stores the transaction entry.

Description

RELATED APPLICATIONS[0001]Applicant claims the benefit of priority of prior, provisional application Ser. No. 61 / 739,685, filed Dec. 19, 2012, the entirety of which is incorporated by reference.FIELD OF INVENTION[0002]This invention relates generally to a storage system and more particularly to creating and using highly available transaction logs in a distributed storage system.BACKGROUND OF THE INVENTION[0003]Enterprise storage systems currently available are proprietary storage appliances that integrate the storage controller functions and the storage media into the same physical unit. This centralized model makes it harder to independently scale the storage systems' capacity, performance and cost. Users can get tied to one expensive appliance without the flexibility of adapting it to different application requirements that may change over time. For small and medium scale enterprise, this may require huge upfront capital cost. For larger enterprise datacenters, new storage applian...

AI Technical Summary

Problems solved by technology

This centralized model makes it harder to independently scale the storage systems' capacity, performance and cost.

Users can get tied to one expensive appliance without the flexibility of adapting it to different application requirements that may change over time.

For small and medium scale enterprise, this may require huge upfront capital cost.

These operate in silos and impose significant management overheads.

Current implementation of a key-value pairs, however, do not lend themselves to supporting multiple data-sets each backed by and independent policy varying in availability, performance, storage placement, storage format, indexing, and consistency guarantees.

In addition, a key-value pair does not handle supporting virtualization that separates data sets and their physical location mapping.

In addition, there is no implementation that provides for named / unnamed writes that co-exists in the same system.

Furthermore, current enterprise storage systems implementations do not optimize on key-value cardinality properties.

For example, the system were to use an “existing” key values store, that system will not be able to optimize on the cardinality property of the system's use case (using a key value store to build a filesystem.)

Images