Quick initialization of data regions in a distributed storage system

Abstract

A method includes flushing a write transaction based on byte-ranges within a file system block. It is determined if the write transaction is a first write transaction to the file system block and whether to flush the received write transaction to a recovery log. If it is determined to flush the write transaction to the recovery log and the received write transaction is the first write transaction: the data associated with the write transaction is appended in the recovery log and byte-ranges remaining in the file system block are recorded, and also recording an associated particular bit value or particular bit pattern that the remaining byte-ranges must be set to before the remaining byte-ranges are accessed or when a recovery sequence for recovering data in the recovery log executes due to failure. The write transaction is replicated. The entire file system block byte-range is marked as committed.

Description

BACKGROUND[0001]Embodiments of the invention relate to buffering and data replication and, in particular, quick initialization using buffering and replicating particular regions in a distributed file system.[0002]Numerous workloads, such as virtual machines (VMs), databases, and accesses to user home directories, send small and synchronous write operations to storage. In addition, many small writes to a file system actually translate into many more small writes to the storage layer to update the recovery log and various metadata structures. Storage controllers typically use non-volatile read and write memory (NVRAM) to buffer these small writes and reduce their latency, but many systems, for example, systems based on a software-defined storage architecture, cannot install such expensive storage devices in every node. This is a particular problem for spinning disk-only based systems because of their poor performance for such operations. While storing data on large numbers of solid-st...

AI Technical Summary

Benefits of technology

[0003]Embodiments of the invention relate to quick initialization using buffering and replicating particular regions in a distributed file system. One embodiment includes a method including flushing a received write transaction from a first memory device to a second memory device on a node in a file system based on one or more byte-ranges within a file system block. In one embodiment, the method determines if the received write transaction is a first write transaction for writing data associated with the received write transaction to the file system block. It is determined to flush the received write transaction to a recovery log that is stored in a non-volatile storage layer of the file system. In one embodiment, if it is determined to flush the received write transaction to the recovery log and the received write transaction is the first write transaction for writing data associated with the received write transaction to the file system block, then the method includes appending the data associated with the received write transaction in the recovery log and recording in the recovery log byte-ranges remaining in the file system block after writing the data associated with the write transaction to the file system block and an associated particular bit value or a particular bit pattern that the remaining byte-ranges must be set to before the remaining byte-ranges are accessed or when a recovery sequence for recovering data in the recovery log executes due to a software or hardware failure. If required, the write transaction as well as information regarding remaining byte-ranges and the associated particular bit value or the particular bit pattern that must be set is replicated in another non-volatile storage layer of another node. Then the entire file system block byte-range is marked as committed.

Problems solved by technology

Storage controllers typically use non-volatile read and write memory (NVRAM) to buffer these small writes and reduce their latency, but many systems, for example, systems based on a software-defined storage architecture, cannot install such expensive storage devices in every node.

This is a particular problem for spinning disk-only based systems because of their poor performance for such operations.

While storing data on large numbers of solid-state drives (SSDs) in every server can improve the situation, it is very costly given the typical capacity requirements in modern data centers.

In addition, naively using SSDs to buffer random writes can severely degrade the lifetime of SSDs.

Images