Method and apparatus for constructing variance-optimized histogram

Abstract

The present invention provides a method and an apparatus for constructing a variance-optimized histogram. The method comprises: setting the number of samples K and the number of histogram barrels B according to the memory size and the query accuracy, and when a new element appears, in order to keep the number of samples at K, optimizing memory space data samples by using the online data sampling method; and according to the optimized memory space data samples, dynamically constructing the variance-optimized histogram. The apparatus comprises an optimization unit and a construction unit. According to the technical scheme provided by the present invention, the influence of the data size and the distribution characteristic is reduced, and the interval retrieval error caused by the data skew or the uneven data distribution can be effectively reduced.

Description

technical field [0001] The invention relates to the field of big data calculation, in particular to a method and device for constructing a variance optimization histogram. Background technique [0002] In the era of big data, streaming data characterized by massive volume and high speed has become a hot research direction. At the same time, the application requirements for real-time processing and analysis of streaming data are also growing explosively. For example, the peak transaction value of Tmall’s “Double Eleven” in 2015 reached 85,900 transactions per second, which was 2.23 times the peak value of 38,500 transactions per second during the “Double Eleven” in 2014. Ant Huabei successfully traded within 1 minute of its launch. The total number of orders has reached 520,000, and the Boeing 737 engine in the flight state generates nearly 20TB of data per hour, such as network monitoring, network traffic analysis, transaction log analysis, stock quotes, transactions, etc., ...

AI Technical Summary

Problems solved by technology

In the prior art, when constructing a variance optimized histogram in a streaming data environment, the required time complexity is O(n·(B / ∈)2logn), and each time a new element is written The complexity is O((B / ∈)logn), which is suitable for ordered stream data, and is limited by memory space, and can only construct data within a specified time window

[0006] The dynamically adjusted approximate variance optimization histogram method in the prior art, each time a new element appears is inserted into the corresponding bucket, and then the bucket is split or merged The sum of the variances of the histogram as a whole is approximately optimal. This method greatly reduces the construction time complexity, but it needs to save all the original data before calculating the variance of the buckets to be split and the buckets to be merged, so it is not conducive to large streams. Dynamically build a variance optimization histogram in the data environment;

[0007] In the prior art, the method of constructing variance optimization histogram with sample data is based on the premise of knowing the distribution of data, and then randomly sampling the continuous stream data, However, the disadvantage is that the accuracy of data analysis is low

Images