Unlock instant, AI-driven research and patent intelligence for your innovation.

Use the processing platform to compute erasure metadata and data layout prior to storage

A technology for processing platform and metadata, applied in the field of parity generation, it can solve the problem of lost computing time, and achieve the effect of reducing the amount of time

Active Publication Date: 2020-04-10
EMC IP HLDG CO LLC
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, almost universally, applications executing on compute nodes are blocked and lose valuable compute time while waiting for the storage system to save written data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Use the processing platform to compute erasure metadata and data layout prior to storage
  • Use the processing platform to compute erasure metadata and data layout prior to storage
  • Use the processing platform to compute erasure metadata and data layout prior to storage

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0017] Illustrative embodiments of the invention will be described with reference to exemplary large-scale computing architectures and associated computing nodes, storage systems, applications, and other processing devices. It should be understood, however, that the invention is not limited to use with the particular illustrative large-scale computing architecture and device configurations shown. Accordingly, the term "large-scale computing architecture" as used herein is intended to be interpreted broadly so as to include, for example, large-scale HPC supercomputers and cloud-based computing and storage systems.

[0018] As noted above, one challenge in large-scale computing architectures when multiple distributed processes write data is the amount of metadata that must be generated, stored, and processed by the storage system. According to one aspect of the present invention, techniques are provided for computing parity metadata, such as erasure codes, using computing capabi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

Techniques are provided for computing data and metadata layout using a processing platform prior to storage in a storage system. An exemplary processing platform includes one or more of a compute node and a burst buffer device. The processing platform communicates over a network with a plurality of computing nodes, wherein a plurality of applications executing on the plurality of computing nodes generate a plurality of data objects; computing erasure metadata for the one or more data objects on at least one of the computing nodes ; and providing erasure metadata and corresponding one or more data objects to the storage system. The processing platform optionally determines a complete set of data objects to be stored, and queries the storage system to determine an expected layout of the complete set of data objects to be stored. The expected layout allows specific handling, eg, for small and large files identified based on predefined criteria.

Description

technical field [0001] The field relates generally to data storage, and more specifically to techniques for generating parity in large-scale computing architectures. Background technique [0002] Large-scale computing architectures, such as high-performance computing (HPC) supercomputers or cloud-based computing systems, typically have a collection of computing nodes dedicated to computing functions and a storage system dedicated to storage functions. However, almost universally, applications executing on compute nodes are blocked and lose valuable compute time while waiting for the storage system to save the written data. Bottlenecks in storage systems can be attributed, for example—especially for streaming data—to the computationally intensive task of generating parity metadata, such as erasure codes and other metadata, as well as the latency of the storage media itself. [0003] As the computing power in the computing nodes of the large-scale computing architecture appro...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F11/10
CPCG06F11/1076G06F3/061G06F3/064G06F3/067H04L67/1097G06F15/167G06F3/0689G06F3/0611
Inventor J·M·班特S·费比斯D·P·J·廷S·特莫瑞J·M·小佩唐G·格赖德
Owner EMC IP HLDG CO LLC