Two-stage-system-based distributed data compression processing method

Abstract

The invention discloses a two-stage-system-based distributed data compression processing method. Based on the conventional mode of compressing data in a server, the data is compressed in interface machines, the compressed data is uploaded to the server after a certain time or when a certain amount of compressed data is obtained, and after the uploaded data is accumulated to a certain amount, the server calls a file processing program to store the data. The compression load of the server is transferred to a plurality of interface machines, so that the memory load of the server is decreased, unnecessary overhead is decreased, the server can give a quicker response to real-time data, and resources of each node in a network are rationally utilized to realize the rational configuration of the resources. By the compression mode of compressing the data in the interface machines and then uploading the compressed data to the server, data transmission in the network is greatly reduced, and network resources are saved. The method for compressing the data in the interface machines and then uploading the compressed data to the server is more suitable for the realization of a real-time database with a super high data volume.

Description

technical field [0001] The application belongs to the technical field of data compression in real-time historical databases, and specifically relates to a distributed data compression processing method based on a two-level system. Background technique [0002] The real-time historical database product is the basic platform of enterprise informatization and the bridge of management and control integration. A large amount of real-time data generated in the production process is one of the valuable resources of process enterprises. These data need long-term storage, fast retrieval, and serve as the basis for production data analysis, data mining, optimization control and optimization management. A typical process enterprise needs to integrate data collection points that usually range from thousands to hundreds of thousands. Since the data at the collection points is dynamically changing and the refresh rate is fast (second level), it is difficult to store such large-scale mass...

AI Technical Summary

Problems solved by technology

Imagine a database with an upper limit requirement of 100,000 label points. The data is updated once per second, and the data volume is 781kb. If all are uploaded to the server for compression, it will take up more than 6M of bandwidth, which greatly consumes network resources.

In addition, the data of each tag point can only be compressed using the compression algorithm after the server has accumulated a certain amount of data, which will greatly increase the memory consumption of the server

Even if the server memory is sufficient, such a large-scale data compression process will bring a great burden to the server CPU.

Images