Storage method and system for kafka's hbase database

Abstract

The invention relates to an Hbase database entering method for kafka. The method includes the steps that S1, all topic data in a kafka cluster is collected and stored in a queue; S2, the corresponding relation between topic and separators in the data, a filtering rule and a database entering rule are configured in a configuration file; S3, according to the corresponding relation, the data is serialized and filtered; S4, configuration information in the configuration file is read, connection with a Hbase database is set up, and the data is constructed into a put object; S5, the put object enters the Hbase. The problem that different topic types in kafka need to be processed individually is solved, the method general and adaptive to all topic methods is provided, the high-performance Hbase database entering method is constructed, the data writing efficiency is greatly improved, output bandwidth is increased, robot network and disk performance are used to the maximum, data safety is ensured to the maximum by means of double-queue design, and data loss is avoided.

Description

technical field [0001] The invention relates to the technical field of computer data processing, in particular to a storage method and system for an Hbase database of Kafka. Background technique [0002] Kafka (Distributed Message Queue) is a high-throughput distributed publish-subscribe messaging system that can handle all action data streams in consumer-scale web sites. Such actions (web browsing, searching and other user actions) are a key factor in many social functions on the modern web. These data are usually addressed by processing logs and log aggregation due to throughput requirements. For log data and offline analysis systems like Hadoop (distributed system framework), because of the limitation of real-time processing, using Kafka for processing is a feasible solution. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and to provide real-time consumption through clustering. [0003] The Hbase data...

AI Technical Summary

Problems solved by technology

[0004] In the process of data collection, it is very common for data to be transferred through Kafka. Kafka designs many topics (topics), and writes data into the Hbase database for different topics. Usually, different methods are required to make the data written into the Hbase database, because The data is different in structure and content, and the topics are different. This makes it necessary to write separate logic codes for different topics to write data, which is time-consuming, laborious, inefficient, and not conducive to later maintenance. Too much logic is expensive to maintain

Images