Metadata management method and apparatus, and distributed file system

A management method and a technology of a management device, which are applied in the field of distributed file storage and can solve problems such as occupying system cpu and disk

Inactive Publication Date: 2017-05-24
ZTE CORP
View PDF5 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Massive files generate hundreds of gigabytes of metadata, and the system upgrade and backup process requires hundreds of gigabytes of disk space for backup, and it takes 1-2 hours for backup; as the system capacity co

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Metadata management method and apparatus, and distributed file system
  • Metadata management method and apparatus, and distributed file system
  • Metadata management method and apparatus, and distributed file system

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0029] figure 1 It is a schematic structural diagram of the metadata management device provided in the first embodiment of the present invention, which is represented by figure 1 It can be seen that in this embodiment, the metadata management device 1 provided by the present invention includes:

[0030] The backup module 11 is used to calculate the differential data between the current metadata and the backed-up metadata by using a differential algorithm when the metadata is backed up;

[0031] The restoration module 12 is configured to restore the current metadata by using the differential data and the backed-up metadata when the metadata is restored.

[0032] In some embodiments, the backed up metadata in the above embodiments includes the latest full backup metadata. In actual use, the change of metadata is gradual. The differential algorithm uses the latest metadata as the calculation basis, which can greatly reduce the data volume of differential data. Of course, those ...

no. 2 example

[0040] figure 2 It is a flow chart of the metadata management method provided in the second embodiment of the present invention, which consists of figure 2 It can be seen that in this embodiment, the metadata management method provided by the present invention includes the following steps:

[0041] S201: When metadata is backed up, use a differential algorithm to calculate differential data between the current metadata and the backed up metadata;

[0042] S202: When restoring metadata, use differential data and backed-up metadata to restore current metadata.

[0043] In some embodiments, the backed up metadata in the above embodiments includes the latest full backup metadata.

[0044] In some embodiments, the differential data in the above embodiments includes the identifier of the metadata that has been backed up, the offset of the differential data, and the differential data.

[0045] In some embodiments, restoring the current metadata using the differential data and th...

no. 3 example

[0051] image 3It is a flow chart of the metadata management method provided by the third embodiment of the present invention, which consists of image 3 It can be seen that in this embodiment, the metadata management method provided by the present invention includes the following steps:

[0052] S301: Set a backup cycle and a full backup threshold.

[0053] When the system is initialized, the backup method is configured, the backup cycle is 3 hours, and the full backup threshold is 1G.

[0054] S302: The time for the first backup is up, and a full backup is performed.

[0055] After the backup period expires, determine the time for the first backup, perform a full metadata backup, and configure the next backup method as differential backup. The generated backup file takes the current system version and timestamp as the backup ID.

[0056] S303: The time for the second backup is up, and differential backup is performed.

[0057] When the next backup cycle arrives, check t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a metadata management method and apparatus, and a distributed file system. The method comprises the steps of calculating differential data of current metadata and backed-up metadata by adopting a differential algorithm during metadata backup; and recovering the current metadata by utilizing the differential data and the backed-up metadata during metadata recovery. By implementing the method and the apparatus, metadata backup and recovery are quickly performed by utilizing a quick differential algorithm and a differential recovery algorithm; an original backup disk space uses order of magnitude reduced to MB from GB; quick backup and recovery of the metadata are realized; and a backup duration of massive metadata of the distributed file system can be shortened to a few minutes from a few hours originally.

Description

technical field [0001] The invention relates to the field of distributed file storage, in particular to a metadata management method and device, and a distributed file system. Background technique [0002] At present, network storage devices are used in products such as CRBT, MS, and WAP gateways. In order to meet the requirements of large capacity, high throughput, and high reliability, with the improvement of the reliability of these products, the price is often exponential. As the level rises, the cost of these devices in the entire system often exceeds 50%. Building a distributed file system on a cheap general-purpose hardware platform has become an inevitable trend in the development of various storage services. [0003] The distributed file system is a general-purpose storage software platform that runs on general-purpose hardware, provides storage platform support for products that require storage services, and provides storage, query, retrieval, and management of mas...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F11/14G06F17/30
CPCG06F11/14
Inventor 郑跃杰
Owner ZTE CORP
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap