HBASE real-time data barrier-free writing processing method based on reading and writing lock algorithm

Abstract

The invention discloses an HBASE real-time data barrier-free writing processing method based on a reading and writing lock algorithm. The method comprises the following steps of 1, banning merging and splitting of HBase through an internal package; 2, defining a global variable; 3, using a constructor; 4, conducting writing processing, wherein a writing lock is added when data is written, the data is stored in a client buffering list, and releasing of the writing lock is completed; 5, conducting reading processing, wherein a reading lock is added when the data is read, the data is submitted to the HBASE server side, and releasing of the reading lock is completed; 6, starting a thread. Compared with the prior art, by the adoption of a JAVA programming language and by utilizing the reading and writing lock algorithm in the JAVA programming language, a client side defines that automatic refreshing of submitting is False, the buffering list obtains the writing lock to store the data, and a timed task is started to obtain the read lock to call an HBase API to achieve submitting of the data. The HBASE real-time data barrier-free writing processing method solves the problem that, since high-concurrency real-time writing data consumption is too slow, stacking of a large amount of data is generated in a queue.

Description

technical field [0001] The invention relates to the application field of computer software programming, in particular to a method for barrier-free writing processing of HBASE real-time data based on a read-write lock algorithm. Background technique [0002] In the process of using HBASE, it is found that when the amount of data written to HBASE is large, it often fails to write in, causing a large amount of data to accumulate in the production queue. However, our HBASE-based application has high requirements for real-time performance. Once HBASE cannot read and write, it will greatly affect the use of the system. The current HBASE multi-threaded writing methods are: [0003] 1) The client side uses setAutoFlush(false) and the size of the cache is XMB. When the table reaches XMB, it can automatically flushCommit; if it is submitted in batches, it can be set to submit every N items; [0004] 2) The client uses setAutoFlush(false) and asynchronous timing refresh submission; ...

AI Technical Summary

Problems solved by technology

[0007] The purpose of the present invention is to provide a solution to the problem that the real-time write data queue consumption of non-relational databases is too slow to generate a large amount of data accumulation under high concurrency, based on the JAVA language read-write lock algorithm to realize the HBASE barrier-free write processing method, through JAVA language generates a read-write lock algorithm, and writes data to HBASE without barriers in real time

Images