Mass sensor data storage system and method based on HBase

A data storage system and data storage technology, applied in the field of big data, can solve problems such as data storage delay, storage hotspot data storage dispersion, and affect the performance of distributed systems, so as to avoid blocking, avoid data storage dispersion, and use well Effect

Inactive Publication Date: 2017-04-05
GUIZHOU UNIV +1
View PDF2 Cites 13 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Using HBase to store large-scale and high-concurrency data has become a relatively mature solution. However, when using HBase to store massive sensor data, due to HBase's own mechanism and the characteristics of large-scale sensor data, data storage delays and storage hot spots will occur. and data storage decentralization problems, which are more prominent when storing massive sensor data, seriously affecting the performance of distributed systems

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Mass sensor data storage system and method based on HBase
  • Mass sensor data storage system and method based on HBase
  • Mass sensor data storage system and method based on HBase

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0032] The present invention will be described in further detail below in conjunction with accompanying drawing, but not as any restriction to the present invention

[0033] Embodiments of the present invention: HBase-based massive sensor data storage system and method, figure 1 Shows the system architecture diagram of the HBase-based massive sensor data storage system of the present invention, refer to figure 1 , the system includes a sensor cluster, a data cache middleware cluster and an HBase distributed database cluster. The sensor cluster includes a sensor sub-cluster, and the data cache middleware cluster includes a data cache middleware child node and a data cache middleware master node.

[0034] The sensor cluster is used to detect data, encapsulate data, and send data. It is the data source of the massive sensor data storage system based on HBase. A sensor cluster is divided into several sensor sub-clusters.

[0035] The data caching middleware cluster is used to c...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a mass sensor data storage system based on HBase. The system and method for storing mass sensor data are designed based on the HBase distributed database cluster, the flow shunting is performed on a data receiving layer to avoid the condition of mass sensor data blocking under a high-concurrence scene; a pre-partitioning and table-establishing operation suitable for storing large-scale sensor data is performed on the HBase distributed database cluster, the data storage hotspot problem caused by the HBase distributed database cluster itself is avoided; an optimizing operation suitable for storing the large-scale sensor data is performed on line units and column names so that the mass sensor data can be continuously stored in the HBase distributed database cluster according to the features of the time sequence, and the scattered data storage problem is avoided. The storage of the mass sensor data based on the HBase is realized. And the system disclosed by the invention is simple and easy to implement, and good in use effect.

Description

technical field [0001] The invention relates to the technical field of big data, in particular to an HBase-based massive sensor data storage system and method. Background technique [0002] With the continuous advancement of science and technology, sensors such as wifi, photosensitive sensors, temperature sensors, etc. are also constantly entering our lives. Therefore, collecting and analyzing sensor data is of great significance for monitoring and predicting environmental changes. When the number of sensors continues to increase, the scale of sensor data will also increase accordingly, which will bring enormous pressure to the massive sensor data storage system. The above sensor data is typical time-series data, which has the characteristics of high concurrency, time serialization, and large overall data size. Aiming at the characteristics of massive sensor data, a data cache middleware cluster written based on the Netty framework is used to receive data requests from sen...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/24552G06F16/2282G06F16/2471
Inventor 黄瑞章刘博伟
Owner GUIZHOU UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap