Data processing method based on distributed storage system

A distributed storage and data processing technology, applied in transmission systems, electrical components, etc., can solve the problems affecting the IO performance of distributed storage systems, waste of disk storage space, and increase IO delay, so as to improve data recovery speed and utilization. rate and the effect of reducing data size

Active Publication Date: 2018-04-20
WUXI CHINAC DATA TECH SERVICE
View PDF13 Cites 25 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Therefore, when the EC-based redundancy technology writes and reads data, the redundancy of the copy is too large
[0009] More importantly, simply increasing the number of K and M will undoubtedly increase the IO delay, and seriously affect the IO performance of CEPH or other types of distributed storage systems, and cause a very large impact on the CPU and memory of the computer. Computational overhead and a huge waste of disk storage space

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method based on distributed storage system
  • Data processing method based on distributed storage system
  • Data processing method based on distributed storage system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0071] The present invention will be described in detail below in conjunction with the implementations shown in the drawings, but it should be noted that these implementations are not limitations of the present invention, and those of ordinary skill in the art based on the functions, methods, or structural changes made by these implementations Equivalent transformations or substitutions all fall within the protection scope of the present invention.

[0072] Before describing each embodiment of the present invention in detail, some technical terms involved in the description and the embodiments are briefly explained and illustrated.

[0073] 1. EC: Erasure-code;

[0074] 2. RADOS: Reliable Autonomous Distributed Object Store;

[0075] 3. RBD: RADOS Block Device;

[0076] 4. RGW: RADOS Gateway;

[0077] 5. CephFS: Ceph Filesystem;

[0078] 6. CRUSH: Controlled Replication Under Scalable Hashing;

[0079] 7. PG: Placement Group;

[0080] 8. OSD: Object Store Daemon;

[008...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data processing method based on a distributed storage system. The method comprises the following steps: defining a compressed configuration item that comprises at least a compression mode, off-line compression start time and off-line compression termination time; when to-be-processed data is written in, using a client or a server, or the client and the server to successively perform chain type compression processing on the to-be-processed data, and determining the time of the server intervening to perform the chain type compression processing only in the off-line compression mode according to the off-line compression start time and the off-line compression termination time; when the to-be-processed data is read, at least enabling the client to perform chain type decompression processing; after the chain type compression processing or the chain type decompression processing is performed, responding to an opposite end device through a network. After the data processing method based on the distributed storage system is used, the bandwidth utilization rate is increased, the memory space is saved, the data reconstruction speed is accelerated, and the data handling capacity of the distributed storage system is improved when a write operation or read operation is performed.

Description

technical field [0001] The invention relates to the technical field of distributed storage, in particular to a data processing method based on a distributed storage system. Background technique [0002] With the development of virtualization, cloud computing and big data, distributed storage systems have become the main way of data storage. In the field of open source distributed storage, distributed storage systems have adopted copy-based redundancy technology or EC technology. [0003] CEPH is an open source unified distributed storage system and one of the most mainstream open source storage projects mentioned above. CEPH is implemented based on the C / S (client / server) architecture. The client can provide external object access interfaces through RADOS, and can also provide high-level application interfaces on top of RADOS. The high-level application interfaces include RBD, RGW and Ceph FS; the server Including OSD, MON and MDS, responsible for data storage, CEPH cluster...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
CPCH04L67/1097H04L69/06H04L67/5651
Inventor 许广彬郑军张银滨李忠文
Owner WUXI CHINAC DATA TECH SERVICE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap