A memory caching method for range query on hadoop

Abstract

The invention discloses a memory caching method oriented to range querying on Hadoop. The memory caching method oriented to the range querying on the Hadoop comprises the following steps that (1) an index is established on querying attributes of Hadoop mass data and is stored on an Hbase; (2) a memory is established on index data of the Hbase to conduct fragment caching, the frequently-accessed index data are selected and stored in the memory, data fragments are fragmented in an initial stage by adopting a fixed length equal dividing method, and the mass data fragments are organized by adopting a skiplist; (3) hit data are queried and recorded according to the data, and the heat of the data fragments is measured by adopting an exponential smoothing method; (4) a memory cache is updated. The memory caching method oriented to the range querying on the Hadoop has the advantages that the structure of combining the skiplist and a collection is adopted, the dynamic adjustment of the fragment boundary of the collection is supported on the structure, the data fragments are made to be adaptive to querying demands, the querying cache hit rate of hot data fragments is improved, the overhead of a querying accessed disk is lowered, and thus the performance of the range querying is improved greatly.

Description

technical field [0001] The present invention relates to information technology, particularly big data query technology, specifically a memory cache method for range query on Hadoop (an open source distributed parallel architecture developed by Apache Foundation). Background technique [0002] Massive data storage and query technology has made great progress in recent years. After NoSQL (non-relational data model) data storage bigtable (Google's non-relational data storage system) proposed by Google (Google), there are currently more than a dozen influential NoSQL database products, such as Hadoop HBase (non-relational data storage system in Hadoop distributed parallel architecture), Yahoo PNUTS (non-relational data storage system of Yahoo), Hadoop Cassandra (formerly Facebook (Facebook) non-relational data storage system) System, later acquired by Hadoop as an open source project), Amazon Dynamo (Amazon's non-relational data storage system), and Hypertable (an open source di...

AI Technical Summary

Problems solved by technology

With the improvement of memory storage performance and the continuous decline in price, the memory caching method commonly used in the database field is also constantly playing a role in the field of massive data query. HBase also provides a caching method for data reading and writing. This caching method is for all The hotspot caching mechanism for read and write requests is based on data blocks (64KB). It cannot create a cache for specific query requirements on specific data, nor can it adjust the cache granularity.

In addition, HBase's existing caching mechanism adopts a simple least recently used (LRU) algorithm for measuring hot data, which does not consider the historical access rules of data, and is prone to frequent data transfer into and out of memory.

Images