Big data query method based on data distribution

Abstract

The invention provides a big data query method based on data distribution, and relates to the technical field of big data query. The method firstly divides data segments of data to be queried, calculates a segment potential distribution function of a whole data segment according to the acceleration ratio to determine the quantitative proportion relationship of each data segment, according to a constraint condition, loads the data to be queried into each data segment, finally, according to the time limit requirement of a user query request, adopts a data segment selection algorithm to determinethe data segments participating in a query, and feeds back a query result, actual recall rate and confidence. The method ensures randomness, performance and approximation evaluation of various queries in a distributed environment, and is compatible with an accurate query, newly added data does not affect the query effect, and good expansibility and maintainability are achieved.

Description

technical field [0001] The invention relates to the technical field of big data query, in particular to a big data query method based on data distribution. Background technique [0002] The high integration of the ternary world of human, machine, and matter has led to the explosive growth of data scale and the high complexity of data models. The world has entered the era of networked big data (Big Data). Faced with such a huge amount of data, how to find the target data within a tolerable time frame is crucial. [0003] Early research on query processing technology mainly focused on the optimization and scheduling of precise queries, and a lot of results have been achieved. However, if the traditional precise query technology is still used today to process massive data, on the one hand, it is difficult to further optimize the query algorithm, and on the other hand, the precise query task will be extremely heavy, resulting in the response time of the entire query exceeding t...

AI Technical Summary

Problems solved by technology

However, if the traditional accurate query technology is still used today to process massive data, on the one hand, it is difficult to further optimize the query algorithm;

In addition, narrowing the search scope (Search Scope) is the main idea to optimize the query. Traditional data partitioning and indexing technologies can accurately narrow the search scope and improve the query hit rate. However, these technologies rely on good division of data value domains and fine data structure, maintaining an accurate data partition and index is expensive in a big data environment

Images