An efficient data transmission method and system between impala and hbase

A data transmission method and a technology of a data transmission system, which are applied in the computer field, can solve problems such as low performance, and achieve high transmission performance and efficient transmission performance

Active Publication Date: 2019-12-03
INST OF COMPUTING TECH CHINESE ACAD OF SCI +1
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

JNI data transfer mode, low performance

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • An efficient data transmission method and system between impala and hbase
  • An efficient data transmission method and system between impala and hbase
  • An efficient data transmission method and system between impala and hbase

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] The implementation process of the present invention will be described in detail below in conjunction with specific embodiments.

[0032] In order to deal with the problems of linear access to Regions in the prior art and the low transmission performance caused by the JNI data transmission mode, the present invention proposes a multi-concurrent transmission mechanism, which implements concurrent data transmission for multiple Regions, and Data is transmitted through the socket to replace the original JNI transmission method.

[0033] Such as figure 2 Shown is a schematic diagram of Impala-HBase data transmission of the present invention. Such as image 3 Shown is a flow chart of an efficient data transmission method between Impala and HBase of the present invention.

[0034] The present invention adds an independent process between Impala and HBase, and the present invention is called HBaseDataEngine. Compared with the technical solution of the existing technology, ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method and a system for high-efficiency data transmission between Impala and HBase. The method comprises the following steps that step 1.query statements sent by the Impala are compiled into query commands, and the number of the query commands is equal to the number of Regions of the HBase; step 2.each query command correspondingly queries one Region of the HBase, all query commands queries the HBase in parallel, and an obtained query result is transmitted to a buffer area in parallel; and step 3.data stored in the buffer area is transmitted to the Impala in a socket way. The method and the system have higher transmission performance, and through adoption of a way of transmitting binary data through socket, a data transmission layer has no expenditure of serialization and deserialization and performance bottleneck, so that the transmission performance is more efficient than the transmission way of JNI.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to an efficient data transmission method and system between Impala and HBase. Background technique [0002] Impala is an open source big data analysis system that can perform data queries on the HBase database. As a data source, HBase provides data to Impala. And Impala is responsible for the calculation of data, that is, SQL query in the usual sense. During this process, Impala needs to access data to HBase. [0003] Currently, Impala uses JNI (Java Native Interface) to call the HBase API for access. JNI is a way for Java and C language to call each other. Through JNI technology, it is possible to call Java code in C language, or call C language code in Java. [0004] As a database, HBase allows each table to have one or more data fragments, that is, one or more Regions. Regions are divided by key-value ranges, there is no intersection between regions, and the data is indep...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/54G06F16/21
CPCG06F9/54G06F16/21
Inventor 程学旗欧陈庚郑天祺张敬亮靳小龙
Owner INST OF COMPUTING TECH CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap