Internet of Things storage system based on NoSQL and distributed file system

Abstract

The invention discloses an Internet of Things storage system based on NoSQL and a distributed file system, and the Internet of Things storage system comprises: an NoSQL interaction module which is responsible for the interaction with an NoSQL database; a distributed file system interaction module which is responsible for interacting with a distributed file system; a time sequence index module which is responsible for storing and indexing time sequence data so as to acquire the data based on time points and time ranges, and is responsible for merging and filing the time sequence data; and a persistence module which is responsible for carrying out special processing on an independent relatively large data object so as to improve the performance. The characteristic of high write-in performance of the NoSQL database and the characteristic that a distributed file system is cheap and easy to expand are effectively utilized to successfully construct the Internet of Things data storage systemwhich is high in performance, low in price, easy to expand and high in availability, and compared with an existing SQL data storage system and a common distributed file system, the performance of theInternet of Things data storage system for storing mass small data of the Internet of Things is remarkably improved.

Description

technical field [0001] The present invention relates to the technical field of IoT storage, in particular to an IoT storage system based on NoSQL and a distributed file system. Background technique [0002] The Internet of Things is a huge network formed by combining massive sensing devices with the Internet. With the development of Internet of Things technology, its application scale continues to expand. Due to the continuous collection and sending of data by massive sensors to the data center, the Internet of Things data presents massive characteristics and explosive growth, forming the big data of the Internet of Things. IoT data not only has a huge storage capacity, but also has the characteristics of various types, variable data size, fast flow, and easy generation of hundreds of millions or even billions of massive small files. Due to the huge challenges in data management, access performance, storage efficiency, etc., its storage has also become an industrial problem...

AI Technical Summary

Problems solved by technology

Since SQL databases need to build complex indexes for querying, when faced with a large number of small files, their index building and traversal queries will cause excessive performance loss, so they cannot be well adapted to IoT storage systems.

Among the emerging NoSQL databases, in-memory databases such as Redis and Memcached have high storage costs. Hbase relies on the cluster environment of the Hadoop family, which is more complicated in operation and maintenance. In comparison, leveldb launched by Google is more suitable as a durable NoSQL. However, as a database embedded in the program, if it is adapted to the Internet of Things program and meets the needs of the Internet of Things program with multiple data centers, large data volumes, and scalability, it requires in-depth secondary development

Images