Data loading system and data loading method for importing Hive mass data into Hbase

Abstract

The invention discloses a data loading system and a data loading method for importing Hive mass data into Hbase, and the data loading method comprises the following steps: step 1, reading to-be-loaded data in a Hive table through a Spark-HFile generation module configured on a big data offline scheduling system, then, an Apache Spark distributed computing engine is adopted to enable data to be loaded in the Hive table to generate a bottom layer HFile file needed by the Hbase table according to mapping configuration of a Hive table field and the Hbase table, and the bottom layer HFile file is written into a specified HDFS directory; and step 2, loading the HFile file in the HDFS directory in the step 1 into an Hbase table on line through an HFile online data loading module configured on a big data offline scheduling system, and providing a user-oriented business query service.

Description

technical field [0001] The invention belongs to the technical field of Hbase databases, and in particular relates to a data loading system and a data loading method for importing Hive mass data into Hbase. Background technique [0002] Hbase is a high-reliability, high-performance, column-oriented, and scalable distributed database. HBase is different from general relational databases. It is a database suitable for structured data storage. Data engineering teams often need to process ETL Import Hive massive data into Hbase to build a user-oriented high-concurrency query service. [0003] The business scenario of Hive importing massive data into Hbase in batches is as follows: the data to be written is located in Hive, and the business needs to periodically import this part of massive data into Hbase to perform random query and update operations. In this scenario, if the scheme of establishing Hive and Hbase mapping tables is adopted, Hbase's Put Api will be called to proces...

AI Technical Summary

Problems solved by technology

In this scenario, if the scheme of establishing Hive and Hbase mapping tables is adopted, Hbase's Put Api will be called to process data writing in the end, which is very likely to bring greater writing pressure to the Hbase RegionServer node, resulting in instability of the Hbase cluster. Even if the RegionServer is down, the solution has the following problems: (1), causing frequent large GC of the RegionServer, resulting in the downtime of the RegionServer node

(2), causing frequent flushing of ReigonServer, and then continuous compression and splitting, affecting the stability of the cluster

(3) Consuming a large amount of CPU disks, bandwidth resources, memory resources and IO resources of the Hbase cluster, creating resource competition with other businesses

This method avoids the cluster resource consumption problem of calling Hbase Put Api, but still faces the following problems: (1) The Map-Reduce engine is implemented based on a multi-process model. During the calculation process, there are multiple IO operations on temporary data. low efficiency

(2) In the process of generating HFile, Map-Reduce will use the default Reduce provided by hbase to determine the number of generated HFile files according to the number of regions of Hbase. In the calculation process, obvious data skew phenomenon is prone to occur, resulting in data loading Prolonged

Images