Real-time industrial process big data compression and storage system and method

Abstract

The invention provides a real-time industrial process big data compression and storage system and method and relates to the technical field of big data analysis. The system comprises a data acquisition module, a log file storage module, a memory cache module, a data compression module, a non-relational database and a rendering device. The data acquisition module is used for acquiring real-time industrial process data and synchronously sending the real-time industrial process data to the memory cache module and the log file storage module, data in the memory cache module is compressed, and the compressed data is stored in the non-relational database. By means of the real-time industrial process big data compression and storage system and method, the purpose of storing real-time industrial process big data can be effectively achieved, the real-time industrial process big data is compressed to the maximum and stored with the non-relational database, the real-time industrial process big data is effectively stored, the huge cost generated for storing the real-time industrial process big data can be reduced for enterprises, and online inquiry, calculation and the like of data can be quickly responded to.

Description

[0001] Technical field: [0002] The invention relates to the technical field of big data analysis, in particular to a real-time industrial process big data compression storage system and method. [0003] Background technique: [0004] With the continuous development of computer technology, big data has emerged as the times require, which has set off a frenzy of digital reform on a global scale. Data has become a valuable resource, and the effective storage and utilization of this resource can enable enterprises to transform and upgrade to intelligent production and intelligent decision-making. At present, real-time big data storage of industrial processes generally adopts real-time databases and relational databases. For industrial enterprises, the data scale is growing at a rapid rate. To ensure that the characteristics of the data itself are not lost and that the stored data can meet the needs of big data analysis Under the premise of this, how to maximize the storage of da...

AI Technical Summary

Problems solved by technology

[0006] At present, the patents on big data compression storage systems mainly include CN1853198A, which involves compressing video images and audio data, using transform coding data such as wavelet transform or DTC, and grouping transform coefficients by most significant bits and / or bit values, And the transmission of them group by group does not involve the compressed storage of real-time big data in industrial processes. As industrial enterprises pay more and more attention to the value of their data, enterprises are gradually building industrial big data collection, storage, and analysis systems. Existing It is difficult for patents to meet the demand for compressed storage of real-time big data in industrial processes

Images