Supercharge Your Innovation With Domain-Expert AI Agents!

Distributed storage method and device

A distributed storage and small file technology, applied in the computer field, can solve the problems of large memory overhead of the name node and low performance of mapping and reduction calculations, and achieve the effects of reducing memory overhead, improving efficiency, and improving computing efficiency

Active Publication Date: 2017-08-15
HUAWEI TECH CO LTD
View PDF5 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0011] The embodiment of the present invention provides a distributed storage method to solve the problems of large memory overhead of the name node and low computing performance of map-reduce when storing small files in the file system

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed storage method and device
  • Distributed storage method and device
  • Distributed storage method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0083] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0084] figure 1 is a schematic flowchart of a distributed storage method according to an embodiment of the present invention, and the method includes S110 to S150. Such as figure 1 As shown, the method is applied to a distributed storage system 100, and the system 100 includes a file system 120, a first thread service 111 and a second thread service 112,

[0085] S110, the first thread service 111 offline merges M small files in the file system 120 into ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

An embodiment of the invention provides a distributed storage method. The method is applied to a distributed storage system. The system comprises a file system, a first thread service and a second thread service, wherein the first thread service combines M small files in the file system into a first file offline according to a first rule; the bytes of the small files are smaller than a predetermined byte; and M is an integer greater than 1. According to the distributed storage method and a distributed storage device provided by embodiments of the invention, through the first thread service and the second thread service, when the small files are stored in the file system, the memory overhead of name nodes and the handle overhead of data nodes can be greatly reduced, and the calculation efficiency of MapReduce can be improved, so that the small file storage efficiency can be improved.

Description

technical field [0001] The invention relates to the computer field, in particular to a distributed storage method and equipment in computer technology. Background technique [0002] Distributed Computing (Hadoop) Distributed File System (Hadoop Distributed File System, referred to as "HDFS") is a distributed file system that stores files in the form of blocks, in which the name node (NameNode) is used to store metadata of files, Including the structure of the directory tree and fast information of file storage, the data node (DataNode) is used to store data blocks. [0003] A small file refers to a file whose size is smaller than the size of a block. HDFS is designed based on Google's Google File System (Google File System, referred to as "GFS") paper to store large files. At present, in HDFS, the default value of the block size is 128MB, and the default value of the block size in the old version is 64MB, and the trend of the block size is getting larger and larger. Howev...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F3/06
CPCG06F3/0604G06F3/067G06F3/061G06F3/0643G06F16/182G06F3/0608G06F11/1435G06F3/0641G06F3/0647
Inventor 张勇蔡艺聪武鸣
Owner HUAWEI TECH CO LTD
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com