Data processing method and device in distributed file storage system

A distributed file and data processing technology, applied in the field of storage, can solve the problems of access performance burden of distributed file storage system, increase in the number of data stripe blocks, etc.

Active Publication Date: 2015-01-07
HUAWEI TECH CO LTD
View PDF4 Cites 49 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In this way, the number of split data stripe blocks increases accordingly when file data is striped. In the case

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device in distributed file storage system
  • Data processing method and device in distributed file storage system
  • Data processing method and device in distributed file storage system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0071] Various exemplary embodiments, features, and aspects of the invention will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures indicate functionally identical or similar elements. While various aspects of the embodiments are shown in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0072] In addition, in order to better illustrate the present invention, numerous specific details are given in the specific embodiments below. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail in order to highlight the gist of the present invention.

[0073] The main components of the distributed file storage system 10 are as attached figure 1 As shown, a plurality of storage servers 101 are included, and the plurality of storage servers 101 are interconnected through a low-latency, high-throughput networ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a data processing method and device 30 in a distributed file storage system. The method includes a client proxy 30 receiving the user data processing request including target file identifier, offset address, file length and other information; the client proxy 30 obtaining redundancy rate information contained in the target file identifier and containing the data strip number N of the distributed file storage system and a verification stripe number M of the distributed file storage system; determining the effective stripe number DSC of the target file according to the offset address and the file length information; determining the actual stripe number N' of the target file based on the effective stripe number DSC and the verification stripe number M; and determining and processing the corresponding stripes according to the actual stripe number N'. The number of the actual stripes generated during striping a target file can be dynamically adjusted according to the size of the file to be processed. In this way, correct target files can always be obtained under any circumstances, and the number of the empty strips in the distributed file storage system 10 can be reduced. Lots of network read-write I/O and disc I/O can be saved under small file conditions, thereby improving the performance of the distributed file storage system 10.

Description

technical field [0001] The invention relates to the field of storage technology, in particular to a data processing method and equipment in a distributed file storage system. Background technique [0002] With the development of computer technology and network technology and the informatization of human life, users have an increasing demand for storage system storage capacity and higher requirements for storage system performance. The storage system has also developed from the computer's own memory to independent storage systems such as storage arrays and network attached storage (Network Attached Storage, NAS), and then to large-scale distributed file storage systems. With the improvement of digitalization, the storage objects have also changed from structured data to unstructured file data such as pictures and micro videos. This puts forward higher requirements on the access performance of file data in the storage system, and improving the access performance of large-scal...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F12/02
CPCG06F2211/1028G06F11/1076G06F16/116G06F16/13G06F16/182G06F16/955G06F16/1727G06F16/1827G06F3/06
Inventor 郭洪星
Owner HUAWEI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap