A Big Data Query Method Based on Data Distribution

Abstract

The invention provides a big data query method based on data distribution and relates to the technical field of big data query. A big data query method based on data distribution. First, the data to be queried is divided into data segments, and the segment potential distribution function of the overall data segment is calculated according to the acceleration ratio to determine the quantity proportion of each data segment; then the data to be queried is divided into segments according to the constraints. The query data is loaded into each data segment; finally, according to the time limit requirements of the user's query request, the data segment selection algorithm is used to determine the data segments participating in the query, and the query results, actual recall rate and confidence are returned. The big data query method based on data distribution provided by the present invention ensures the randomness, performance and approximation evaluation of various queries in a distributed environment, is compatible with precise queries, and the new data will not affect the query effect, and has good Scalability and maintainability.

Description

technical field [0001] The invention relates to the technical field of big data query, in particular to a big data query method based on data distribution. Background technique [0002] The high degree of integration of the ternary world of man, machine, and things has triggered an explosive growth in data scale and a high degree of complexity in data patterns. The world has entered the era of networked Big Data (Big Data). Facing such a huge amount of data, how to find the target data within a tolerable time range is crucial. [0003] Early research on query processing technology mainly focused on the optimization and scheduling of precise queries, and a lot of results have been obtained. However, if the traditional accurate query technology is still used to process massive data, on the one hand, it is difficult to further optimize the query algorithm; In addition, narrowing the search scope (Search Scope) is the main idea to optimize the query. Traditional data partition...

AI Technical Summary

Problems solved by technology

However, if the traditional accurate query technology is still used today to process massive data, on the one hand, it is difficult to further optimize the query algorithm;

In addition, narrowing the search scope (Search Scope) is the main idea to optimize the query. Traditional data partitioning and indexing technologies can accurately narrow the search scope and improve the query hit rate. However, these technologies rely on good division of data value domains and fine data structure, maintaining an accurate data partition and index is expensive in a big data environment

Images