Small file management service (SFMS) system based on MapFile and use method thereof

A technology for managing services and small files, applied in file systems, file system management, special data processing applications, etc., can solve problems such as excessive memory and occupation of master nodes, so as to reduce memory usage, improve efficiency, and reduce the difficulty of use Effect

Inactive Publication Date: 2014-02-05
XIDIAN UNIV
View PDF3 Cites 15 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The purpose of the present invention is to provide a small file management service SFMS system based on MapFile and its use method, so as to solve the problem that the master node will occupy too much memory when storing a large number of small files on the distributed file system HDFS on Hadoop

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Small file management service (SFMS) system based on MapFile and use method thereof
  • Small file management service (SFMS) system based on MapFile and use method thereof
  • Small file management service (SFMS) system based on MapFile and use method thereof

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0035] see figure 1 , as can be seen from the figure, the server is composed of SFMS and Hadoop clusters. Users send requests to SFMS to manage files, such as reading files, writing files, or deleting files, etc., and then SFMS converts these requests into Hadoop The operation of MapFile in the cluster realizes the management of small files in this way.

[0036] see figure 2 , as can be seen from the figure, the data file stores the Key-Value pair, while th...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a small file management service (SFMS) system based on MapFile and a use method thereof. A small file merging strategy based on time is adopted, a plurality of index files are kept in a memory, small file reading efficiency is further improved, and finally massive small files are merged into a finite number of MapFile, the whole process is packaged into a service, a jar package is provided for client writing staff, and the client writing staff can use the system and the method to conduct operation of storing, reading and deleting the small files massively and the like conveniently. The system and the method aim at solving the problem that when the massive small files are stored in a hadoop distribution file system (HDFS), master node memory occupying rate is too high.

Description

technical field [0001] The invention relates to a small file management service SFMS system based on MapFile in Hadoop developed by the Apache foundation and a usage method thereof. Background technique [0002] Hadoop is a distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without knowing the underlying details of the distribution. Make full use of the power of the cluster for high-speed computing and storage. Hadoop implements a distributed file system, HDFS for short. HDFS is highly fault-tolerant and designed to be deployed on inexpensive hardware. And it provides a high transfer rate to access application data, suitable for those applications with very large data sets. [0003] Hadoop is designed to handle large files. It stores incoming large files on different data blocks of different datanodes, and records the metadata of large files on the namenode. When operating on a file, first visit the namenode, loc...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F16/11G06F16/182
Inventor沈沛意董洛兵张亮宋娟孙庚泽马汉炜
OwnerXIDIAN UNIV