Mass sensor data storage system and method based on HBase

Abstract

The invention discloses a mass sensor data storage system based on HBase. The system and method for storing mass sensor data are designed based on the HBase distributed database cluster, the flow shunting is performed on a data receiving layer to avoid the condition of mass sensor data blocking under a high-concurrence scene; a pre-partitioning and table-establishing operation suitable for storing large-scale sensor data is performed on the HBase distributed database cluster, the data storage hotspot problem caused by the HBase distributed database cluster itself is avoided; an optimizing operation suitable for storing the large-scale sensor data is performed on line units and column names so that the mass sensor data can be continuously stored in the HBase distributed database cluster according to the features of the time sequence, and the scattered data storage problem is avoided. The storage of the mass sensor data based on the HBase is realized. And the system disclosed by the invention is simple and easy to implement, and good in use effect.

Description

technical field [0001] The invention relates to the technical field of big data, in particular to an HBase-based massive sensor data storage system and method. Background technique [0002] With the continuous advancement of science and technology, sensors such as wifi, photosensitive sensors, temperature sensors, etc. are also constantly entering our lives. Therefore, collecting and analyzing sensor data is of great significance for monitoring and predicting environmental changes. When the number of sensors continues to increase, the scale of sensor data will also increase accordingly, which will bring enormous pressure to the massive sensor data storage system. The above sensor data is typical time-series data, which has the characteristics of high concurrency, time serialization, and large overall data size. Aiming at the characteristics of massive sensor data, a data cache middleware cluster written based on the Netty framework is used to receive data requests from sen...

AI Technical Summary

Problems solved by technology

[0004] Using HBase to store large-scale and high-concurrency data has become a relatively mature solution. However, when using HBase to store massive sensor data, due to HBase's own mechanism and the characteristics of large-scale sensor data, data storage delays and storage hot spots will occur. and data storage decentralization problems, which are more prominent when storing massive sensor data, seriously affecting the performance of distributed systems

Images