Variance optimization histogram construction method and device based on Spark Streaming

Abstract

The invention relates to a variance optimization histogram construction method and device based on Spark Streaming. The method comprises the steps that Spark Streaming is utilized to perform online sampling on streaming data; a variance optimization histogram is dynamically constructed according to online sampling data; and the variance optimization histogram is dynamically updated by use of newly added data, and the variance optimization histogram is dynamically constructed again according to the newly added data. Through the technical scheme, the high-precision approximate variance optimization histogram can be constructed by scanning data once within a limited memory space.

Description

technical field [0001] The invention relates to the field of big data computing, in particular to a method and device for constructing a variance optimization histogram based on Spark Streaming. Background technique [0002] With the rapid development of the Internet, Internet of Things, cloud computing, and communication technologies, the explosive growth of data has brought new development opportunities and challenges to many industries. Efficient computing of massive data has become the focus of current research. The computing mode of big data can be divided into two modes: batch computing and streaming computing. Since streaming data has the characteristics of persistence, timeliness, burstiness, and unknown data distribution, compared with batch computing of offline data, streaming data The online processing technology is not yet perfect, and with the development of technology, the demand for efficient calculation of streaming data and statistical analysis and other app...

AI Technical Summary

Problems solved by technology

[0004] The traditional algorithm based on dynamic programming needs to traverse the data set multiple times, so the time and space complexity are large; there is also a method that can construct a variance optimization histogram for any data set in sub-linear time, but the method can only be used for offline Data cannot satisfy the construction of variance-optimized histograms in streaming data environments; for the premise of limited memory space in streaming data environments, the academic community has also proposed a way to use sample data to construct variance-optimized histograms, but the construction of this method The premise is that the data distribution must be known in advance, and the continuously arriving flow data can be randomly sampled according to the obtained data distribution information; in addition, there is currently a dynamically adjusted approximate variance optimization histogram method, which inserts each newly arrived element into the corresponding In the buckets, the sum of the overall variance of the histogram is approximately optimal by splitting and merging the buckets. The advantage of this method is that it greatly reduces the time complexity of constructing a variance-optimized histogram, but the disadvantage is that all original data need to be saved. Therefore, this method is not conducive to dynamically constructing variance optimization histograms in a limited-space streaming big data environment; at present, under the framework of distributed computing, a method using MapReduce The calculation framework constructs an approximate variance optimization histogram method for the data in the probability database, but this method can only be calculated for offline data, and cannot be quickly constructed for online streaming data, and the current popular streaming data computing platform does not provide calculation variance Methods for Optimizing Histograms

Images