Large-scale industrial data compression storage method, system and medium

Abstract

The invention provides a large-scale industrial data compression storage method, system and medium, including: step 1: configuring different data acquisition systems according to the type of data source, and extracting the data collected by the data acquisition system through interface operation; step 2 : Define the conversion chain, and temporarily convert the format of different types of data extracted into Avro format through the data cleaning plug-in; Step 3: Compress the data in Avro format with the GPL protocol, and the compression format is snappy, and create the following in the distributed file system Parquet is a data set in a storage format, which stores compressed data. The invention can define conversion chains and compression and storage formats for any type of data, greatly improving the data processing speed and data compression ratio of the computing platform.

Description

technical field

[0001] The present invention relates to the technical field of data compression storage, in particular to a large-scale industrial data compression storage method, system and medium. Background technique

[0002] With the vigorous development of new infrastructure, more and more traditional industrial enterprises have begun to use Internet technology to improve productivity, among which data is the most critical. In the traditional Internet, there are more and more data in big data processing, and many companies will back up two copies of data. This results in a waste of disk.

[0003] Patent document CN108304472A (application number: 201711455790.2) discloses a data compression storage method and a data compression storage device. The data compression method includes the following steps: a segmentation step, which divides the original data into multiple fields; and a compression step, based on Depending on the data content, different compression strategies...

Images