Method and device for storing data to Hbase based on Siddhi

A technology for storing data and data, applied in database models, relational databases, structured data retrieval, etc., can solve the problems of stream data storage technology and high environmental requirements, and achieve low learning costs, simple storage operations, and easy-to-understand structures Effect

Inactive Publication Date: 2021-08-06
CHENGDU SEFON SOFTWARE CO LTD
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The purpose of the present invention is to provide a method and device for storing data to Hbase based on Siddhi to solve the problem of high technical and environmental requirements for existing stream data storage

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0028] A method of storing data to Hbase based on Siddhi. First, implement a custom HbaseSink by inheriting Siddhi's Sink abstract class, and then store stream data in batches into Hbase through Siddhi; the specific process is as follows:

[0029] The first step is to preset the parameters in the entire SiddhiQL statement, including hbase.zookeeper.quorum, namespace, tablename, columnfamily required in Hbase, and also add flushing trigger condition parameters: flush.count and time.out, through Time and data volume to control the timing of flashing.

[0030] The second step is to inherit Siddhi's Sink abstract class and implement the core abstract method. This is a mature technology in this field, so I will not repeat it here, and the general custom Sink framework will be built.

[0031] The third step is to realize the writing of HbaseClient. Specifically, use the API provided by Hbase to connect to the Hbase cluster, and write data to the table under the specified namespace. ...

Embodiment 2

[0035] A device for customizing HbaseSink based on Siddhi includes a memory: for storing executable instructions; a processor: for executing the executable instructions stored in the memory, and implementing a method for storing data to Hbase based on Siddhi.

[0036] The invention is used to store streaming data processed by Siddhi into Hbase, and store data in Hbase by customizing the Sink type of Siddhi, thereby realizing continuous storage of large-scale streaming data.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method and device for storing data to an Hbase based on Siddhi. The problem that in the prior art, existing streaming data storage has high requirements for the technology and the environment is mainly solved. The method for storing the data to the Hbase based on the Siddhi comprises the steps of: presetting parameters in a whole SiddhiQL statement; then, inheriting a Sink abstract class of Siddhi, realizing a core abstract method in the Sink abstract class, and completing the construction of a user-defined Sink framework; and storing the stream data into the Hbase in batches by triggering conditions. According to the scheme, the purpose of reducing technical requirements and environment requirements required by stream data storage is achieved, and the method has very high practical value and popularization value.

Description

technical field [0001] The present invention relates to the technical field of stream data storage, in particular to a method and device for storing data to Hbase based on Siddhi. Background technique [0002] The current continuous storage of streaming data into Hbase is to pull data from Kafka through spark-streaming, and then do simple data cleaning and finally store it in Hbase; this method saves streaming data into Hbase in real time, and the streaming data storage efficiency is high. However, there are higher requirements for the technology and processing environment, which at least need to provide complete Kafka and spark environment support, and not only need to consider how the target data is published to Kafka, but also need to be supported by spark programming. Contents of the invention [0003] The purpose of the present invention is to provide a method and device for storing data to Hbase based on Siddhi, so as to solve the problem of high technical and enviro...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25G06F16/28
CPCG06F16/25G06F16/284
Inventor 周刚雷友奇杨尧胥莉君
Owner CHENGDU SEFON SOFTWARE CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap