Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method for moosefs to realize multilevel storage of files based on Tachyon

A multi-level storage and file technology, which is applied in the field of big data and cloud storage, can solve the problems that the performance of writing data is difficult to meet the delay of streaming computing and high network delay, so as to reduce learning costs, reduce overhead, and avoid memory overflow Effect

Inactive Publication Date: 2017-03-29
SHANDONG INSPUR COMML SYST CO LTD
View PDF2 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

There is a high network delay between the local computing cluster and the remote storage cluster, and frequent remote data exchange becomes a major bottleneck in the entire stream processing process
[0015] 2. The design of MooseFS is disk-based, and its I / O performance, especially the performance of writing data, is difficult to meet the delay required by streaming computing

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for moosefs to realize multilevel storage of files based on Tachyon
  • Method for moosefs to realize multilevel storage of files based on Tachyon

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0042] The content of the present invention is described in more detail below:

[0043] In the present invention, by introducing the Tachyon memory distributed file system into the MooseFS file system, the files stored in the tachyon can be accessed in the cluster at the speed of accessing the memory. A middleware between frameworks. The main responsibility is to put those files that do not need to be stored in the distributed disk file system into the distributed memory file system to achieve shared memory and improve efficiency. At the same time, it can reduce memory redundancy, GC time, etc. It can also be easily connected with hadoop's map reduce and spark computing models.

[0044] MooseFS is implemented based on fuse. After setting the mount point on the MFS client, Tachyon can operate MooseFS like a Linux general file system, which brings great convenience to deployment and implementation.

[0045] step description

[0046] 1 Download and install Tachyon

[0047] $...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a method for a moosefs to realize multilevel storage of files based on a Tachyon and belongs to the fields of cloud storage and big data. Through introducing the Tachyon memory distributed file system into the MooseFS file system, the files in the tachyon can be accessed in a cluster at the memory access speed, and the Tachyon is a middleware between a distributed file storage at the lowest level and various computing frames at the upper level. The main responsibilities are that the files which do not need to fall to a distributed disk file system fall into a distributed memory file system to realize memory sharing, so that the efficiency is improved. Meanwhile, the memory redundancy, the GC time and the like can be reduced. The method also abuts against map reduce and spark computing models of hadoop easily.

Description

technical field [0001] The invention relates to the fields of cloud storage and big data, in particular to a method for moosefs to realize multi-level storage of files based on Tachyon. Background technique [0002] MooseFS is a network distributed file system. It spreads data across multiple servers, but for users, they only see one source. MFS, like other unix-like file systems, contains a hierarchical structure (directory tree), stores file attributes (permissions, last access and modification time), and can create special files (block devices, character devices, pipes, sockets) ), symbolic links, hard links. [0003] Advantages of MooseFS: [0004] 1. It can be expanded online, and the system architecture is extremely scalable. [0005] 2. Easy to deploy. Universal file system, which can be used without modifying the upper layer application (supports fuse). However, some old versions of the kernel, such as 5.4 or below, may need to add a fuse module [0006] 3. Th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30G06F12/0875
CPCG06F16/182G06F12/0875G06F16/172
Inventor 葛天成曹苗苗宋育千
Owner SHANDONG INSPUR COMML SYST CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com