HDFS-based heterogeneous Internet-of-things big data storage and reading system and method

Abstract

The invention relates to an HDFS-based heterogeneous Internet-of-things big data storage and reading system and method. The system comprises a data query server, a client and a plurality of data storage centers, the data query server is respectively connected with the client and each data storage center; wherein each data storage center comprises a data acquisition module, a data processing serverand a Hadoop cluster, the data acquisition module is connected with the data processing server, the data processing server is connected with the Hadoop cluster, and the Hadoop cluster is connected with the data query server; wherein the Hadoop cluster comprises an HDFS distributed file system; wherein the storage parameters of each Hadoop cluster are stored in the data query server. Compared withthe prior art, the data query server is arranged outside the plurality of Hadoop clusters to store the IP address and the HDFS file information of each cluster, the barrier of the traditional data architecture is broken through, and a user can query or download the data of the plurality of Hadoop clusters through the data query server without accessing each Hadoop cluster in sequence.

Description

technical field [0001] The invention relates to the field of big data storage and reading, in particular to a system and method for storing and reading big data of heterogeneous Internet of Things based on HDFS. Background technique [0002] On the one hand, with the rapid development of network technology and communication technology, human society has entered the era of big data, and people are paying more and more attention to the processing and analysis of big data; on the other hand, with the advancement of smart cities, various departments The Internet of Things system generates a large amount of data. The department refers to the organization that manages an Internet of Things system (such as a smart transportation system), such as transportation, environmental protection, water conservancy, meteorology, etc. The Internet of Things system in a smart city is often a heterogeneous Internet. [0003] Hadoop is an open source distributed computing platform. Hadoop is base...

AI Technical Summary

Problems solved by technology

[0006] However, in the IoT system, limited by the geographical environment and cost, the IoT data of multiple departments are stored in different data centers. Even if the Hadoop architecture is used for data storage, the data of each department is stored in different Hadoop clusters.

If users want to read the data of different departments, they must visit the data centers of each department in turn to read the data. The data of different departments are not interoperable, which brings obstacles to the comprehensive analysis and use of data.

Images