Distributed real-time compression method for multiple data streams

Abstract

The invention relates to a data stream compression technology, in particular to a distributed real-time compression method for multiple data streams, which comprises the following steps of: firstly, carrying out coarse-grained online fragmentation on data streams by utilizing a correlation relationship; using a greedy strategy for constructing a coarse-grained time series data flow online fragmentation algorithm, wherein on one hand, the correlation of time series data flows in fragments is considered, compression errors and grouping dynamics are guaranteed, and on the other hand, it is guaranteed that the fragment length is maximum, that is, the compression rate is considered; carrying out fine-grained clustering on the data streams in the fragments; aiming at data characteristics in the fragments, providing a time sequence-oriented segment clustering algorithm from the perspective of unsupervised learning, so that not only can semaphore with approximate readings in the fragments be subjected to fine-grained grouping, but also core line segments and noise segments in the groups can be marked, and the later representative sequence selection process is accelerated; and finally, selecting a representative data stream sequence in the cluster for storage and compression. By means of the method, the compression effect and efficiency are improved.

Description

technical field [0001] The invention belongs to the technical field of data stream compression, in particular to a multi-data stream-oriented distributed real-time compression method. Background technique [0002] With the vigorous development of sensor technology, more and more fields begin to monitor the data in the system (such as air quality detection, traffic status monitoring, climate change detection, etc.). However, in order to provide near real-time monitoring, sensors must be sampled at a high frequency, which results in unprecedented large-scale streaming time-series data. More and more scholars have proposed that these data must be effectively aggregated, stored and retrieved in order to be better mined and utilized. Intuitively, these sequence structures have time correlation or inter-sequence correlation. Using this feature to compress time series can reduce storage overhead on the one hand, and on the other hand, will not affect the performance of monitoring ...

AI Technical Summary

Problems solved by technology

However, in order to provide near real-time monitoring, sensors must be sampled at a high frequency, which produces unprecedented large-scale streaming time-series data

Images