Data loading system and data loading method for importing massive Hive data into Hbase

Abstract

The invention discloses a data loading system and a data loading method for importing Hive mass data into Hbase. The data loading method comprises the following steps: step 1, reading the data to be loaded in the Hive table through a Spark-HFile generation module configured on a big data offline scheduling system, and then using an Apache Spark distributed computing engine to map the data to be loaded in the Hive table with Hbase tables according to Hive table fields, generate the underlying HFile file required by the Hbase table, and write the HFile file into a specified HDFS directory; step 2, loading the HFile file in the HDFS directory in step 1 into the Hbase table online through an HFile online data loading module configured on the big data offline scheduling system, and providing user-oriented business query services.

Description

Technical Field

[0001] The present invention belongs to the technical field of Hbase database, and in particular relates to a data loading system and a data loading method for importing Hive mass data into Hbase. Background Art

[0002] Hbase is a highly reliable, high-performance, column-oriented, scalable distributed database. HBase is different from general relational databases. It is a database suitable for structured data storage. Data engineering teams often need to import massive Hive data processed by ETL into Hbase to build high-concurrency query services for users.

[0003] The business scenario of Hive importing massive data to Hbase in batches is as follows: the data to be written is located in Hive, and the business needs to regularly import this part of the massive data into Hbase to perform random query and update operations. In this scenario, if the solution of establishing a Hive and Hbase mapping table is adopted, the Hbase Put API will eventually be called to process data writing, which is very likely to bring great writing pressure to the Hbase RegionServer node, causing the Hbase cluster to be unstable or even the RegionServer to crash. This solution has the following problems: (1) It causes frequent large GC in the RegionServer, resulting in RegionServer node crashes. (2) It causes the ReigonServer to flush frequently, and then continuously compress and split, affecting the stability of the cluster. (3) It consumes a large amount of CPU disk, bandwidth resources, memory resources and IO resources of the Hbase cluster, causing resource competition with other businesses.

[0004] In the scenario of writing large amounts of data to Hbase, the Put API solution is generally abandoned and the Bulkload solution is adopted. The Bulkload solution provided by Hbase natively is implemented based on the MapReduce computing engine. The MapReduce engine is a multi-process computing engine. Compared with the multi-threaded distributed computing engine Spark, it consumes large computing resources and has low computing efficiency. Therefore, choosing the Spark engine to implement Hbase's Bulkload massive data writing can improve computing efficiency and reduce computing resource consumption.

[0005] For example, patent document CN106897450A discloses a method for quickly importing massive data into Hbase based on HDFS (distributed file system): It describes a solution for importing massive data from HDFS into Hbase based on Map-Reduce. This method uses the Map-Reduce engine to generate HFiles, and according to the number of pre-partitions of hbase, batches of generated Hbase underlying data files are moved to achieve batch import of data into Hbase. This method avoids the problem of cluster resource consumption when calling the Hbase Put API, but still faces the following problems: (1) The Map-Reduce engine is implemented based on a multi-process model. During the calculation process, there are multiple IO operations of temporary data, and the calculation efficiency is low. (2) During the generation of HFiles, Map-Reduce will use the default Reduce implementation provided by hbase to determine the number of generated HFile files according to the number of regions of Hbase. During the calculation process, obvious data skew is likely to occur, resulting in longer data loading time. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide a data loading method that can import massive Hive data into Hbase with stable data import and fast data import speed.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A data loading system for importing Hive mass data into Hbase comprises a Hive database and an Hbase database, and is characterized in that it also comprises a Spark-HFile generation module and an HFile online data loading module. The Spark-HFile generation module is used to read the data to be loaded in the Hive table of the Hive database, and after generating the underlying HFile file required by the Hbase table according to the mapping configuration between the Hive table field and the Hbase table in the Hbase database, the HFile file is written into a specified HDFS directory; the HFile online data loading module is used to load the HFile in the HDFS directory specified by the Spark-HFile generation module into the Hbase table online to provide business query services.

[0009] A data loading method for importing massive Hive data into Hbase comprises the following steps: step 1, reading the data to be loaded in the Hive table through a Spark-HFile generation module configured on a big data offline scheduling system, and then using an Apache Spark distributed computing engine to configure the data to be loaded in the Hive table according to the mapping configuration of the Hive table field and the Hbase table, generating the underlying HFile file required by the Hbase table, and writing it into a specified HDFS directory; step 2, loading the HFile file in the HDFS directory in step 1 into the Hbase table online through an HFile online data loading module configured on the big data offline scheduling system, and providing user-oriented business query services.

[0010] In this way, the loading method adopted uses the Spark computing engine to accelerate the ETL process of generating HFile, and adopts the method of mixing multiple region data in one HFile, avoiding the problem of fixed reduce number of native Hbase interface, avoiding data skew, improving computing efficiency and reducing the use of computing resources. The online data loading module adopts the data block loading method of Hbase, directly moving the generated HFile data file to the data directory of the Hbase table, avoiding the problem of excessive pressure on Hbase nodes caused by writing large quantities of data into Hbase using Put API, and providing stable and efficient Hbase cluster massive data loading capabilities.

[0011] Furthermore, in step 1, the HFile file generation algorithm is as follows: S1, use the spark.sql operator to read Hive data and convert it into DataFrame, use the dtypes method of DataFrame to obtain Hive's schema information and convert it into spark's broadcast variables; S2, convert the DataFrame of Hive data into a resilient distributed dataset RDD, apply the flatMap operator on the RDD and introduce the schema broadcast variable at the same time, generate the Hbase KeyValue RDD class required by HFile in the calculation logic, and globally sort the RDD; S3, obtain the corresponding Table class according to the hbase table metadata in the configuration file, directly introduce the Table class into the HFile output file type to configure the HFile generation task; S4, write the RDD sorted by KeyFamilyQualifier to the HDFS directory. In this way, the algorithm used does not use the Reduce setting method provided by Hbase itself, gets rid of the data skewness of generating HFile, and avoids the efficiency bottleneck of writing data from the same Region into one HFile.

[0012] Furthermore, in step 2, the generated HFile files are loaded online into the corresponding HBase table through the LoadIncrementalHFiles component of HBase. When loading HFile online, the upper limit of HFile import for a single region and a single family is set. In this way, after setting the import upper limit, the stability of data import can be guaranteed.

[0013] Furthermore, after obtaining the metadata of the Hbase table, the HFile online data loading module scatters the data into different regions. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a system architecture diagram of an embodiment;

[0015] Figure 2 This is a diagram of the HFile storage data directory planning of the embodiment.

[0016] Figure 3 The following is a flow chart of the HFile file generation algorithm of the embodiment. DETAILED DESCRIPTION

[0017] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0018] Example:

[0019] like Figure 1 As shown, the data loading system for importing massive Hive data into Hbase provided by this embodiment includes a Hive database, an Hbase database, a Spark-HFile generation module and an HFile online data loading module. The Spark-HFile generation module is used to read the data to be loaded in the Hive table in the Hive database, and generate the underlying HFile file required by the Hbase table according to the mapping configuration between the Hive table field and the Hbase table in the Hbase database, and then write the HFile file into the specified HDFS directory; the HFile online data loading module is used to load the HFile in the HDFS directory specified by the Spark-HFile generation module into the Hbase table online to provide business query services.

[0020] The above modules are all mounted on the big data offline scheduling system in the form of task nodes. Every morning, the data to be loaded in the Hive table is loaded online into the Hbase table to provide high-concurrency query services for users.

[0021] Specifically, when loading massive amounts of Hive data into Hbase, the system administrator builds the Spark-HFile generation module and the data online loading module as a task flow in the big data distributed scheduling system in the form of task nodes, and configures the daily task scheduling cycle. The scheduling system will start the data migration workflow at regular intervals every morning.

[0022] like Figure 2 As shown, the data loading method for importing massive data from Hive into Hbase includes the following steps: Step 1, reading the data to be loaded in the Hive table through the Spark-HFile generation module configured on the big data offline scheduling system, and then using the Apache Spark distributed computing engine to map the data to be loaded in the Hive table with the Hbase table according to the Hive table field and the Hbase table, generating the underlying HFile file required by the Hbase table, and writing it into the specified HDFS directory; Step 2, loading the HFile file in the HDFS directory in step 1 into the Hbase table online through the HFile online data loading module configured on the big data offline scheduling system, and providing user-oriented business query services.

[0023] Specifically, the HFile generation module based on the Apache Spark distributed computing engine reads the data to be loaded in the Hive table according to the hive.table.name and hive.partition.condition parameters configured in the configuration file. The Hive data of type DataFrame is converted into RDD constructed by Hbase KeyValue objects through ETL, and then the KeyValue RDD is converted into HFileOutputFormat2 and written to the following directory of HDFS: hdfs: / / nameservice:8020 / tmp / hbase / hfile / namespace_table / yyyy-MM-dd.

[0024] In step 1, the HFile file generation algorithm is as follows Figure 3As shown in the figure: S1, use the spark.sql operator to read Hive data and convert it into DataFrame, use the dtypes method of DataFrame to obtain Hive's schema information and convert it into spark's broadcast variables; S2, convert the DataFrame of Hive data into a resilient distributed dataset RDD, apply the flatMap operator on the RDD and introduce the schema broadcast variable at the same time, generate the Hbase KeyValue RDD class required by HFile in the calculation logic, and globally sort the RDD; S3, obtain the corresponding Table class according to the hbase table metadata in the configuration file, and directly introduce the Table class into HFileOutputFormat2 to configure the HFile generation task; S4, write the RDD sorted by KeyFamilyQualifier to the HDFS directory.

[0025] The HFile generation module is a Spark offline computing application. The sample script for submitting a task is shown below:

[0026] #! / bin / bash

[0027] spark2-submit \

[0028] --name "HiveToHbase HFile generation" \

[0029] --class com.eights.hbase.bulkload.HiveToHbase \

[0030] --master yarn \

[0031] --deploy-mode cluster \

[0032] --files bk.properties \

[0033] --executor-memory 10G \

[0034] --num-executors 30 \

[0035] --driver-memory 5g \

[0036] --executor-cores 1 \

[0037] batch-app-cluster-ops-1.0-SNAPSHOT.jar bk.properties.

[0038] In step 2, the generated HFile file is loaded online into the corresponding HBase table through the LoadIncrementalHFiles component of HBase. When loading HFile online, (please provide additional information on what settings are used) set the upper limit of HFile import for a single region and single family. In order to ensure the stability of data import, a larger value is generally configured, such as 102400. The sample execution script is as follows:

[0039] Hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles\

[0040] –D hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily=102400 \

[0041] hdfs: / / target-cluster:8020 / tmp / hbase / hfile / namespace_table / yyyy-MM-ddnamespace_table.

[0042] After obtaining the metadata of the Hbase table, the HFile online data loading module will scatter the data into different regions. Each region of Hbase has a start and end line. After obtaining the metadata of the hbase table, the HFile online data loading module will determine which region the HFile file needs to be loaded into based on the start and end lines of the HFile file to be loaded. After determining the region, the corresponding RegionServer will officially provide services after successfully loading the HFile.

[0043] Using a data loading technology for importing massive Hive data into Hbase in this embodiment, using the computing resources limited by the HFile generation module sample task, it takes a total of 12 minutes to generate an HFile file corresponding to a 370 million Hive table. At the same time, it avoids the problem of RegionServer node downtime caused by Hbase large-scale data Put. This embodiment provides a stable and efficient data loading technology for quickly importing massive Hive data into Hbase.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Although the applicant has described the present invention in detail with reference to the preferred embodiments, ordinary technicians in this field should understand that those modifications or equivalent substitutions of the technical solution of the present invention without departing from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A data loading method for importing massive Hive data into Hbase, comprising the following steps: Step 1, reading the data to be loaded in the Hive table through the Spark-HFile generation module configured on the big data offline scheduling system, and then using the Apache Spark distributed computing engine to map the data to be loaded in the Hive table with the Hbase table according to the Hive table field and Hbase table, generating the underlying HFile file required by the Hbase table, and writing it into a specified HDFS directory; the HFile file generation algorithm is as follows: S1, using the spark.sql operator to read the Hive data and convert it into a DataFrame, using the dtypes method of the DataFrame to obtain the schema information of the Hive and convert it into a broadcast variable of the spark; S2, converting the DataFrame of the Hive data into a resilient distributed dataset RDD, applying the flatMap operator on the RDD and introducing the schema broadcast variable at the same time, generating the Hbase KeyValue required by the HFile in the calculation logic RDD class, globally sort the RDD; S3, obtain the corresponding Table class according to the hbase table metadata in the configuration file, introduce the Table class and the sorted RDD generated in S2 into the HFile output file type to configure the HFile generation task; S4, use the HFile generation task configured in S3 to write the sorted RDD generated in S2 to the specified HDFS directory; Step 2, load the HFile file in the HDFS directory in step 1 into the Hbase table online through the HFile online data loading module configured on the big data offline scheduling system, and provide user-oriented business query services.

2. The method for loading massive Hive data into Hbase according to claim 1, characterized in that: In step 2, the generated HFile files are loaded online into the corresponding HBase table through the LoadIncrementalHFiles component of HBase. When loading HFile online, the upper limit of HFile import for a single region and a single family is set through the HBase client parameter hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily.

3. The method for loading Hive mass data into Hbase according to claim 1 or 2, characterized in that: After obtaining the metadata of the Hbase table, the HFile online data loading module scatters the data into different regions.

Images