A method and device for hbase-based strongly ordered queue operation

Abstract

The present invention provides a method and device for HBase-based strongly ordered queue operation, which acquires queue operation requests sent by multiple clients, wherein the queues targeted by the queue operation requests are located in the HBase distributed database, if more Queue operation requests at the same time and for the same queue, perform queue lock competition for the multiple queue operation requests, and perform queue operation corresponding to the queue operation request on the queue for the queue operation request with successful lock. After the queue operation is completed, the queue lock is released, and the above steps are repeated until the execution of the multiple queue operation requests is completed, so as to realize the HBase-based strongly ordered queue operation. Compared with the existing technology, the present invention has the advantages of high performance, strong order, high availability, support for massive equipment and data is not easily lost, and can be applied to cloud Internet of Things access services, with good performance, high reliability, and horizontal expansion Strong compatibility, satisfying the access of a large number of users and devices.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a technology of HBase-based strong ordered queue operation. Background technique [0002] On the public cloud, IoT services are often multi-tenant, each user has a different device, and each subscribing device has its own message queue. As the number of users and devices increases, the number of queues and the size of queue data will also increase dramatically, often becoming a performance bottleneck. [0003] Most of the existing distributed queue solutions are implemented with Redis or Kafka. First of all, from a distributed perspective, both support cluster mode and can form a distributed system. Secondly, from the perspective of queues, Redis supports queue structures, and Kafka's topics can be used as queues to meet the orderly writing of messages. Finally, from the perspective of multi-tenancy, logical division can be made. For example, user+client can be used t...

AI Technical Summary

Problems solved by technology

[0004] However, Redis and Kafka can meet small-scale requirements to a certain extent, but cannot meet larger-scale and higher-reliability scenarios

First of all, from the perspective of Redis, it is a memory-based database. For mass storage, the cost will be quite high. Its cluster mode is master-slave, and there is a risk of data loss. For example, the master node is down during master-slave replication. , or both master and slave nodes are down, data will be inconsistent and lost

Secondly, from the perspective of Kafka, there is a performance bottleneck problem. Because Kafka stores topic and other information in zookeeper, when the number of topics reaches a certain scale, that is, the number of znodes is large, the limitation of zookeeper will cause Kafka performance to drop sharply

It can be seen that the current public cloud IoT service providers cannot support massive access scale, nor can they achieve high availability and high reliability

Images