Method and system for extracting and converting data between Hbase and Hdfs

Abstract

The invention relates to a method and a system for extracting and converting data between an Hbase and an Hdfs. The method comprises the following steps of: acquiring a to-be-processed data table name and processing parameters; according to the to-be-processed data table name and the processing parameters, processing a to-be-processed data table; and inserting the processed data table into an Hdfs. According to the method and the system, historical data can be extracted and stored in the Hdfs, and the data in the Hdfs also can be recovered into the Hbase when the history data is needed; a tool just has such a function; data extraction, backup and storage can be realized through configuration modification according to different environments; and furthermore, normal use of the generated Hbase is not influenced.

Description

technical field [0001] The invention relates to Hadoop big data clusters, in particular to a data extraction and conversion method and system between Hbase and Hdfs. Background technique [0002] In the context of the big data era, the processed data is calculated at the T-level and PB-level. Traditional technologies have gradually been unable to handle data of this order of magnitude. New technologies such as Hadoop clusters and Hbase have emerged as the times require. For example: the cloud detailed list storage widely used now, the detailed list is stored in the Hbase database, but you need to back up and store the historical detailed list, you can use this tool to store the Hbase data in Hdfs, and you can also use this tool if necessary Restore the data in Hdfs to the Hbase database. [0003] In comparison, traditional relational databases have the following disadvantages: [0004] (1) The two-dimensional tabular data model adopted by the relational database cannot eff...

AI Technical Summary

Problems solved by technology

[0004] (1) The two-dimensional tabular data model adopted by the relational database cannot effectively handle multi-dimensional data, and cannot effectively handle semi-structured and unstructured massive data in Internet applications, such as Web pages, emails, audio, video, etc.

[0005] (2) The performance of high concurrent read and write is low;

[0008] Relational databases can barely cope with tens of thousands of SQL queries, but hard disk I / O often cannot bear tens of thousands of SQL write data requests

[0009] (3) The supporting capacity is limited;

[0011] (a) Taking Facebook as an example, 135 billion (unconfirmed) user updates are stored in a month. For a relational database, SQL queries in a table with 135 billion records are extremely inefficient or even impossible. enduring

[0012] (b) Another example is the user login system of a large Web site or IM, such as Tencent and MSN, with hundreds of millions of accounts at every turn, and relational databases are also difficult to handle

[0013] (4) The scalability and availability of the database are low;

[0014] When the number of users and visits of an application system are increasing day by day, the traditional relational database has no way to expand the performance and load capacity simply by adding more hardware and service nodes like WebServer

Images