HBase-based big data storage and retrieval method and system

Abstract

The invention discloses an HBase-based big data storage and retrieval method and system. A line key is generated through a defined RowKey expression with an HBase Thrift client based on a data file field mapping table, and data stored in lines is imported into an HBase database. On the premise of keeping consistence, multi-feature values of data objects are added according to a plurality of combination ways to form the line key; the multi-feature values and ordinary column value data construct HBase data lines; the HBase data lines are stored in a plurality of HBase data tables according to different line key construction ways; a fuzzy result set can be rapidly obtained according to matching of certain feature values in the line key during multi-feature value data retrieval; and filter filtration is further performed on the fuzzy result set to obtain a final accurate result set. A research result can be suitable for a big data conversion-storage process from different types of data files to a destination HBase database; high universality is achieved; line key storage data is formed according to a multi-feature value combination way; a rapid data retrieval interface can be provided; and the aim of rapidly retrieving is fulfilled.

Description

technical field [0001] The invention belongs to the technical field of big data storage, and relates to a method and system for distributed storage, transformation and retrieval of big data. Its goal is to realize the transformation of data files stored in rows into HBase distributed database storage and to quickly retrieve and access the big data in HBase. Background technique [0002] There are few related literatures on the conversion and storage of data files stored by row to HBase. There are usually three ways to migrate and integrate different types of data into HBase: one is to use the HBase interface to write a special program to realize data docking; the other is to use bulk load, etc. The tool completes data import; the third is to write a MapReduce program to import data to HBase. These three methods have problems such as poor versatility, limited use environment, and high operational complexity. It is necessary to develop a general data extraction and transforma...

AI Technical Summary

Problems solved by technology

Since the source data file structure, field names, and types are different, and the storage methods of data items are also different, it is difficult to implement such data conversion and import using existing HBase import tools such as bulk loader

Usually to solve similar problems, the programmer must write a specific program to complete the data extraction and import. However, this program is generally only applicable to the import of a specific source data file to the destination table. If the source data file type or the destination table changes then the program no longer applies

Images