Adaptive compression method and system for distributed file system

A technology of distributed files and compression methods, applied in special data processing applications, instruments, electrical digital data processing, etc., can solve the problems of heavy system load, lack of ability to automatically select the optimal compression algorithm, etc., to achieve good adaptability sexual effect

Active Publication Date: 2016-06-29
INST OF COMPUTING TECH CHINESE ACAD OF SCI
View PDF5 Cites 13 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0009] Through the comparison of the existing transparent compression mechanism and adaptive compression model, it is found that no adaptive compression model is used in the distributed file system, although some compression models may be applicable to the distributed file sys...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Adaptive compression method and system for distributed file system
  • Adaptive compression method and system for distributed file system
  • Adaptive compression method and system for distributed file system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0041] The following are the overall steps of the present invention, as follows:

[0042] like Figure 6 As shown, the present invention proposes an adaptive compression method of a distributed file system, comprising:

[0043] Step 1, setting the compressed file format to form a compressed data stream, the compressed data stream is composed of header information and a plurality of data blocks, wherein the header information is used to determine whether the compressed data stream has been compressed;

[0044] Step 2, receiving the file to be compressed, compressing the file to be compressed according to the compressed file format to generate the compressed data stream;

[0045] Step 3, setting an index file, wherein the index file is composed of multiple records, each record maintains data information of the data in the compressed data stream, and the index file is used to quickly locate the compressed data stream.

[0046] The data information includes the location of the f...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention provides an adaptive compression method and system for a distributed file system, and relates to the field of distributed system file compression. The method comprises the steps of: setting a compressed file format, and forming a compressed data stream, wherein the compressed data stream consists of head information and a plurality of data blocks, and the head information is used for determining whether the compressed data stream is compressed; receiving a to-be-compressed file, and compressing the to-be-compressed file according to the compressed file format so as to generate the compressed data stream; and setting an index file, wherein the index file consists of a plurality of records, and each record maintains data information of data in the compressed data stream, and the index file is used for rapidly positioning the compressed data stream. The method and system provided by the present invention are capable of improving compression efficiency and saving compression time.

Description

technical field [0001] The invention relates to the field of distributed system file compression, in particular to an adaptive compression method and system of a distributed file system. Background technique [0002] With the advent of the data age, the amount of data processed by the Internet is increasing. In order to achieve high reliability, the current distributed file system generally adopts a multi-copy strategy. In a large-scale cluster, this will bring a huge amount that cannot be ignored. Storage overhead. At the same time, for systems or applications on distributed file systems, such as distributed databases, distributed data warehouses, MapReduce frameworks or other applications, redundant data may also be generated, which will make the data expansion rate even higher. High, I / O performance has become the bottleneck of the system, and the existing distributed file system is difficult to meet the requirements of high performance, high reliability and low storage o...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/1744G06F16/182
Inventor 查礼王锐坚王超
Owner INST OF COMPUTING TECH CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap