A general high-performance data writing method for hbase database
A data writing, high-performance technology, applied in the field of big data writing to HBase database, can solve problems such as blocking HBase's regionserver client, affecting storage efficiency, and regionserver downtime, avoiding concurrent security risks and avoiding queue data. The effect of blocking and avoiding data interference
Active Publication Date: 2018-01-09
北京中交兴路车联网科技有限公司
View PDF5 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0003] In terms of data writing into the Hbase database, most of the existing technologies adopt the common producer-consumer mode, because the production and consumption threads snatch synchronization locks, especially under multi-threading, which seriously affects the storage efficiency.
Another existing technology is to use the import function of the tool that comes with the Hbase database to import data, which is less efficient
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0009] This general high-performance data writing method for Hbase database uses a data distribution mechanism to evenly distribute multiple copies of data. The data in different tables has different data processing objects. The HTable object array is a local variable of this object; One corresponds to an HTable object array; each piece of data is written into a buffer, and then the HTable object array reads the written buffer for storage operation; the HTable object array adds data to the used buffer during the process of writing data. When the thread lock is applied, the data not written into the buffer bypasses the buffer with the thread lock and writes a new buffer.
[0010] The present invention uses a data distribution mechanism to evenly distribute multiple data, and the data in different tables has different data processing objects, and the receivers (HTable object arrays) of the distributed multiple data are all local variables of this object, thus ensuring the table l...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a universal high-performance data writing method for an Hbase database, and is capable of avoiding the data interference and concurrence potential risk, avoiding the queue data block caused by slow writing speed of the HTable data, and ensuring the continuous output of the queue data. The universal high-performance data writing method for the Hbase database comprises the steps of using the data distribution mechanism to distribute multiple groups of data averagely, wherein the data in the different lists has different data processing objects, and all HTable object arrays are the local variable of this object; corresponding to one HTable object array by each group of the data one to one; writing each group of the data to multiple buffer regions, and reading the written buffer region to perform the in-stockroom operation by the HTable object array; adding a thread lock to the used buffer region in the process of writing the data by the HTable object array, and bypassing the buffer region added with the thread lock and writing a new buffer region by the data unwritten in the buffer region.
Description
technical field [0001] The invention relates to the technical field of computer data processing, in particular to a general high-performance data writing method for an Hbase database, which is mainly used for writing large data into the Hbase database. Background technique [0002] HBase is a distributed, column-oriented open source database. Using HBase technology, a large-scale structured storage cluster can be built on a cheap PC Server. HBase is an open source implementation of Google Bigtable, similar to Google Bigtable using GFS as its file storage system, HBase using Hadoop HDFS as its file storage system; Google runs MapReduce to process massive data in Bigtable, HBase also uses Hadoop MapReduce to process massive data in HBase ; Google Bigtable uses Chubby as a collaborative service, and HBase uses Zookeeper as a counterpart. [0003] In terms of data writing into the Hbase database, most of the existing technologies adopt the common producer-consumer model, becaus...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
CPCG06F16/258
Inventor 曹宇
Owner 北京中交兴路车联网科技有限公司