Metadata updating method and device based on column storage of distributed file system as well as host

A technology for distributed files and file metadata, applied in the field of data processing, can solve the problems of not being able to provide a method for dynamic modification of metadata, consuming huge computing resource overhead and time overhead, etc.

Active Publication Date: 2015-08-26
HUAWEI TECH CO LTD
View PDF4 Cites 20 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, the purpose of the present invention is to solve the problem that in the existing Append-Only file system, an effective method for dynamically mo

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Metadata updating method and device based on column storage of distributed file system as well as host
  • Metadata updating method and device based on column storage of distributed file system as well as host
  • Metadata updating method and device based on column storage of distributed file system as well as host

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0105] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0106] See figure 1 The embodiment of the present invention provides a method for updating metadata based on columnar storage in a distributed file system. The method includes:

[0107] Step 110: Obtain the metadata to be updated in the data table of the distributed file system, divide the data records of the data table into multiple row groups by row, and convert the data table into a global file metadata and multiple row group files .

[...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

A method, device and host for updating metadata stored in columns in a distributed file system, the method comprising: obtaining the metadata to be updated in a data table in the distributed file system; segmenting the data records in the data table by row into a plurality of row groups; converting the data table to one global file metadata and a plurality of row group files (110), the row group files comprising actual data blocks, data index blocks, local metadata blocks, metadata index blocks and file footers; determining whether the metadata to be updated belongs to the global file metadata (120); if not, then updating the local metadata (130); adding the updated local metadata blocks, the metadata index blocks and the file footers to the a plurality of row group files according to the updated local metadata(140). The method realizes dynamic updating of the metadata, greatly saves the execution time of such updating operation and the required calculation resources.

Description

technical field [0001] The present invention relates to the field of data processing, and more specifically relates to a method, device, and host for updating metadata based on columnar storage of a distributed file system. Background technique [0002] Append-Only file system, that is, the file system only supports file read or append write operations, unlike traditional file systems that support in-place file modification operations. This type of file system has the characteristics of high write performance and easy consistency of multiple copies, and is widely used in large-scale distributed storage systems. Typical representative examples include GFS (Google File System, Google Distributed File System), HDFS (Hadoop Distributed File System, Hadoop Distributed File System), etc. In the Append-Only file system, columnar storage has unique advantages over traditional row-based storage. In columnar storage, data records are divided into columns and stored independently, so ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F3/0638G06F3/061G06F3/067G06F16/164G06F16/13G06F16/182G06F16/196G06F16/1805
Inventor 邓利群朱俊华
Owner HUAWEI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap