Data import method and device based on Hbase database, electronic equipment and medium

Abstract

The invention relates to big data, and discloses a data import method based on an Hbase database. The method comprises: receiving an original data set, and unifying the data format of the original data set by using a preset computing framework to obtain a target data set; creating an identification field of each piece of target data in the target data set, and performing hash processing on the identification field of each piece of target data to generate a target field of each piece of target data; according to the target field, creating a data pre-storage partition of an Hbase database, and storing target data corresponding to the target field into the data pre-storage partition; and loading the target data in the data pre-storage partition to a data table of the Hbase database by utilizing a preset data loading method. The invention also relates to a blockchain technology, and the original data set can be stored in a blockchain node. According to the method, the timeliness and the success rate of data import based on the Hbase database can be improved.

Description

technical field [0001] The present invention relates to the field of big data technology, in particular to a method, device, electronic equipment and computer-readable storage medium for data import based on Hbase database. Background technique [0002] The Hbase database is a NoSQL database based on Apache Hadoop. It is a distributed and column-oriented dynamic schema database that can support real-time reading and writing and random access to very large-scale data sets. [0003] At present, there are two main methods for data import based on the Hbase database: method 1, directly calling the Put interface of Hbase to import data; method 2, relying on the MapReduce framework to perform data batch conversion and then import data. [0004] However, by calling the Put interface of Hbase to import data, it is easy to occupy too much Hbase database resources, which will affect the timeliness of data import; by relying on the MapReduce framework to perform data batch conversion a...

AI Technical Summary

Problems solved by technology

[0004] However, by calling the Put interface of Hbase to import data, it is easy to occupy too much Hbase database resources, which will affect the timeliness of data import; by relying on the MapReduce framework to perform data batch conversion and then import data, it will rely too much on the MapReduce framework , once the MapReduce framework fails, it will affect the success rate of data import

Images