Hbase database entering method and system for kafka

A database and data writing technology, applied in database design/maintenance, digital data processing, structured data retrieval, etc., can solve the problems of high logic maintenance cost, disadvantageous later maintenance, low efficiency, etc., to improve efficiency, avoid The effect of losing and increasing the output bandwidth

Active Publication Date: 2016-05-25
北京中交兴路车联网科技有限公司
View PDF4 Cites 25 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In the process of data collection, it is very common for data to be transferred through Kafka. Kafka designs many topics (topics), and writes data into the Hbase database for different topics. Usually, different methods are required to make the data written into the Hbase database, because

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Hbase database entering method and system for kafka
  • Hbase database entering method and system for kafka

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0035] Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0036] like figure 1 As shown, the present invention provides a kind of storage method for the Hbase database of kafka (distributed message queue), it is characterized in that, comprises the steps:

[0037] A storage method for the Hbase database of kafka is characterized in that, comprising the steps:

[0038] S1: Collect the data of all topics (topics) in the kafka cluster and save them in the queue;

[0039] S2: Configure the corresponding relationship between the topic and the delimiter, filter rule, and storage rule in the data in the configuration file;

[0040] S3: Serialize the data according to the corresponding relationship, and filter the data;

[0041] S4: read the configuration information in the configuration file, create a connection with the Hbase database, and construct the data into a put object;

[0042] S5: Store the put object in...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to an Hbase database entering method for kafka. The method includes the steps that S1, all topic data in a kafka cluster is collected and stored in a queue; S2, the corresponding relation between topic and separators in the data, a filtering rule and a database entering rule are configured in a configuration file; S3, according to the corresponding relation, the data is serialized and filtered; S4, configuration information in the configuration file is read, connection with a Hbase database is set up, and the data is constructed into a put object; S5, the put object enters the Hbase. The problem that different topic types in kafka need to be processed individually is solved, the method general and adaptive to all topic methods is provided, the high-performance Hbase database entering method is constructed, the data writing efficiency is greatly improved, output bandwidth is increased, robot network and disk performance are used to the maximum, data safety is ensured to the maximum by means of double-queue design, and data loss is avoided.

Description

technical field [0001] The invention relates to the technical field of computer data processing, in particular to a storage method and system for an Hbase database of Kafka. Background technique [0002] Kafka (Distributed Message Queue) is a high-throughput distributed publish-subscribe messaging system that can handle all action data streams in consumer-scale web sites. Such actions (web browsing, searching and other user actions) are a key factor in many social functions on the modern web. These data are usually addressed by processing logs and log aggregation due to throughput requirements. For log data and offline analysis systems like Hadoop (distributed system framework), because of the limitation of real-time processing, using Kafka for processing is a feasible solution. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and to provide real-time consumption through clustering. [0003] The Hbase data...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/21
Inventor 曹宇余效伟肖赞李旭阳
Owner 北京中交兴路车联网科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap