Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Management method for metadata in distributed file system

A technology of distributed files and management methods, applied in the field of metadata management in distributed file systems, can solve problems affecting system availability, loss of metadata files, long recovery time, etc., to improve system availability and enhance security , strong practical effect

Active Publication Date: 2014-04-30
GUANGDONG INSPUR BIG DATA RES CO LTD
View PDF4 Cites 21 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In order to reduce the load, the method of separating metadata and data is generally used to persist metadata generated files to the local disk, so that the system startup time will be linearly related to the system size. When the file system size exceeds 50 million, the startup time It will take tens of minutes, and the recovery time of the system from the failure is extremely long, which seriously affects the availability of the system. When the metadata file is accidentally damaged, the system will be completely unavailable, and various complex technical means need to be adopted to improve the metadata. file security, but metadata files are still at risk of loss

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Management method for metadata in distributed file system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0025] Example: as attached figure 1 As shown, the whole file system is composed of 4 metadata servers forming a metadata server cluster and 5 data servers. The metadata cluster shares the metadata storage work of dirA and dirB, and the two are mutually backed up.

[0026] Each of the five data servers stores 100,000 files. When the data server is started, it traverses the managed disks in a wide range, and only traverses the two-level directory, that is, the traversal ends, and the traversal results are sent to the metadata server to complete the system startup. If the level of the file directory is relatively shallow, all concentrated in the first two levels, the traversal will be terminated after a certain period of time, and the metadata will be sent to the metadata server to control the startup time of the system.

[0027] After the metadata arrives at the metadata server cluster, the metadata cluster stores the metadata in groups. For example, there are two directories ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a management method for metadata in a distributed file system. The method includes the special management processes that the system is divided into a metadata server providing metadata services and a data server, and a user can acquire a file system view through the metadata sever to obtain a directory structure and a file list of the file system; when the user visits a metadata cluster, if the metadata to be accessed are not found in the metadata cluster, data are provided to the user through the data server; when a memory occupied by a metadata cache exceeds a threshold value, part of the memory is released so as to control memory occupancy of the metadata server. Compared with the prior art, the management method for the metadata in the distributed file system has the advantages that single-point problems are solved by the metadata by the aid of a cluster mode, the metadata are stored in the data server, further the system is started rapidly, and system availability is improved.

Description

technical field [0001] The invention relates to the data security technology of cluster computers, more specifically to the management method of metadata in the distributed file system. Background technique [0002] With the development of information technology, the advent of cloud computing and the era of big data, people need to process more and more data. Generally, distributed file systems are used to store massive amounts of data. At present, there are various problems in distributed file systems. , such as the single point of failure of the metadata server, this problem is generally solved by dual-machine hot backup. In order to reduce the load, the method of separating metadata and data is generally used to persist metadata generated files to the local disk, so that the system startup time will be linearly related to the system size. When the file system size exceeds 50 million, the startup time It will take tens of minutes, and the recovery time of the system from ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/13G06F16/182
Inventor 闫宁
Owner GUANGDONG INSPUR BIG DATA RES CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com