Distributed file system architectural method supporting mass data access

A distributed file and massive data technology, applied in the field of distributed file system architecture, can solve the problems of not supporting random reading and writing of files, excessive dependence on the database, and the existence of single point of failure, so as to overcome single point of failure and reduce system complexity Sexuality, the effect of reducing time consumption

Active Publication Date: 2014-08-27
SOUTH CHINA UNIV OF TECH
View PDF3 Cites 38 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

But its single master node is easy to become a bottleneck, and there is a single point of failure
MogileFS supports the reading and writing of a large number of small files

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed file system architectural method supporting mass data access
  • Distributed file system architectural method supporting mass data access
  • Distributed file system architectural method supporting mass data access

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0056] The hardware network structure adopted by the distributed file system supporting massive data access described in this embodiment is as follows figure 1 As shown, it is a two-tier system architecture, which specifically includes a client and several servers. Each server includes a name node (NameNode) and a data node (DataNode). Similar to the traditional C / S architecture, the client accesses files through the API In the system, the nodes in the cluster communicate with each other through Ethernet, and each node is responsible for maintaining the routing table, metadata, and file data. The client can realize the following operations: A. Connect to any node; B. Connect to a specific server. The client can operate on files by connecting to any node that has registered the service.

[0057] The architecture method described in this embodiment is based on a distributed hash table, and access nodes are obtained by performing hash mapping on file paths. This system adopts ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a distributed file system architectural method supporting mass data access. The method is based on a distributed Hash table, and access nodes are obtained by conducting Hash mapping on a file path. By the adoption of the fully distributed type non-centralized architectural design, new nodes can join a cluster after several times of communications. A Kademlia algorithm is adopted for addressing among the nodes, and a routing table is divided and the distance between the nodes is obtained through exclusive-or operation to achieve skip of the most adjacent nodes. A leader is selected through a PaxosLease algorithm to handle the mapping operation on the nodes so as to solve the consensus problem. Block storage of fixed size is conducted on actual data of files, and redundant backup is conducted on the multiple nodes, so that the requirements for safety and distributed calculation are met. When an architectural system generated according to the architectural method processes mass files, the processing efficiency can be remarkably improved, and the good effect can be achieved in the environment with the low latency requirement.

Description

technical field [0001] The invention relates to the field of distributed file system research, in particular to an architecture method of a distributed file system supporting mass data access. Background technique [0002] With the development of Internet technology, "cloud computing" is increasingly valued by people. It is a new oriented technology formed by the integration of traditional technologies such as distributed computing, parallel computing, utility computing, network storage, virtualization, and load balancing. User's service product concept. And "cloud storage" is one of the cloud services closest to ordinary netizens. [0003] In the early distributed file system, files and their metadata information were not backed up redundantly. Once one of the servers fails, the files stored on the server will not be available. And as the number of files increases, the system becomes larger, making it difficult to scale and manage. Modern distributed file systems pay mor...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/134G06F16/182
Inventor 董敏金泽豪毕盛
Owner SOUTH CHINA UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap