An online aggregation method for multi-table joining based on Markov chain

Abstract

The invention provides an on-line aggregation method of multi-table connection based on Markov chain, includes two phases: sample creation and online gathering, At that sample creation stage, Create ahierarchical sample of the original dataset in conjunction with load characteristics, The hierarchy is based on the grouped column set in the query load, so that the probability of the column set appearing in the load and the probability of the grouped column set being covered in the load are maximized. Based on the determined grouped column set and the distribution of the index, the join order of each table is determined, and the hierarchy sample is created at the running start of the Markov chain. In the online aggregation stage, the multi-table join query statement submitted by the user isparsed, the samples with the lowest query cost are dynamically selected for stratified sampling, the size of samples extracted from each sample layer is determined, and the query result and the confidence interval are further estimated.

Description

technical field [0001] The invention relates to a big data analysis method, and mainly relates to an online aggregation method for multi-table connections based on Markov chains. Background technique [0002] Social media, mobile devices, and sensors continue to generate massive amounts of data at an unprecedented rate. Exploring the value behind these data has become a matter of great concern to the industry and academia. However, complex data analysis tasks run slowly on massive data. The timeliness and value of the analysis results are greatly reduced, and it becomes a bottleneck for data-driven tasks to play a role. Ad hoc interactive data analysis plays an important role in the fields of decision support, trend analysis, and data visualization, and has become one of the urgent problems to be solved in the field of big data analysis. Online aggregation continuously processes part of the sample data, so that statistically significant estimation results can be returned in...

AI Technical Summary

Problems solved by technology

[0005] In view of the above problems, the present invention proposes an online aggregation method for multi-table connections based on Markov chains, which converts the multi-table connection processing process into a traversal walk process on the Markov chain, and creates a branch at the starting point of the walk based on the model. Layer samples, and perform unbiased estimation and confidence interval calculation for the sampling method, effectively solving the problem of inaccurate result estimation and slow convergence of confidence intervals caused by connection load or data skew

Images