Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

File processing method based on distributed file system, system, and client

A distributed file and client technology, applied in the transmission system, electrical components, etc., can solve the problems of occupying disk storage space and low file read and write performance

Inactive Publication Date: 2012-12-26
SHENGQU INFORMATION TECH SHANGHAI
View PDF0 Cites 49 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The embodiment of the present application provides a file processing method, system, and client based on a distributed file system, so as to solve the existing problem of reading and writing distributed files according to the file layout allocation table, which causes disk storage space to be occupied, and file reading and writing poor performance

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • File processing method based on distributed file system, system, and client
  • File processing method based on distributed file system, system, and client
  • File processing method based on distributed file system, system, and client

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0040] see figure 2 , which is the flow chart of the first embodiment of the file processing method based on the distributed file system of the present application:

[0041] Step 201: The virtual node-based consistent hash ring is stored in the client.

[0042] For data storage nodes in the distributed file system, each storage device on it corresponds to at least one virtual node on the consistent hash ring according to its storage capacity, and each virtual node saves the data of the storage device to which the virtual node belongs Device identification and usage status, assigning a virtual node value to each virtual node based on the total number of virtual nodes. Further, every two adjacent virtual nodes do not belong to the same storage device, so as to reduce the possibility of selecting virtual nodes belonging to the same storage device during the write operation.

[0043] Step 202: the client sends a file writing request to the metadata node.

[0044] When the data...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a file processing method based on a distributed file system, the system, and a client. The method comprises the following steps: the client sends a file writing request to a meta node; the client receives a file identifier, a data block identifier of each data block, the copy quantity of data blocks and a selectivity factor from the meta node; the client performs hashing operation on the file identifier of the file, the data block identifiers of the data blocks and the selectivity factor to obtain a hashing value; the client finds a consistent hashing ring according to the hashing value, obtains the value of a minimal virtual node greater than the hashing value and clockwise selects virtual nodes consistent with the copies in quantity from the virtual node on the consistent hashing ring; and the client sends the data blocks to different storage devices for storage when the selected virtual nodes meet the pre-condition. According to the invention, the occupation of the storage space of the meta node is reduced, the location speed during data writing can be improved, and the load balance of the system is implemented.

Description

technical field [0001] The present application relates to the technical field of computer storage, in particular to a file processing method, system and client based on a distributed file system. Background technique [0002] Distributed File System (Distributed File System) means that the physical storage resources managed by the file system are not necessarily directly connected to the local node, but are connected to the node through a computer network. The design of the distributed file system is based on the client / server model. see Figure 1A , is a schematic diagram of a typical distributed file system architecture, mainly including clients, metadata nodes and data storage nodes. Among them, the client (Client) is usually deployed on the business application server, and the business application program sends data access requests to the distributed file system cluster through the client; the data storage node (chunk node) is used to store file data, and each data The...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): H04L29/08H04L29/06
Inventor 俞强华邓明潘佳伟王本结
Owner SHENGQU INFORMATION TECH SHANGHAI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com