Storage and retrieval method based on massive meteorological data

Abstract

The invention discloses a storage and retrieval method based on massive meteorological data for solving the problem of traditional concentrated data storage and single-point query. By means of a Hadoop platform, reliable massive data storage and rapid massive data retrieval are achieved by building a secondary index for a distributed non relational database Hbase and inputting data into a cloud platform through conversion and migration. The method comprises the steps of filtering data, defining a corresponding sheet format in the Hbase, building the secondary index, conducting data inputting according to conditions, and conducting data retrieval according to conditions. According to the storage and retrieval method, real-time data query can be achieved, the high cost generated by existing massive data storage and maintenance is avoided, and the real-time massive meteorological data query can be more economically and efficiently achieved on the premise that the sensitive data security is ensured.

Description

technical field [0001] The invention belongs to the technical field of cloud storage, and in particular relates to a storage and retrieval method based on massive meteorological data. Background technique [0002] Cloud computing technology originated in the business world. Due to its powerful ability to process big data, it has become a trend in the development of computer technology, and thus has attracted the attention of the industry and academia. With the development of cloud computing, the importance and value of cloud storage are becoming more and more significant. Measuring the availability of the system is usually expressed by the ratio of the normal service time of the system to the total running time, which is calculated as follows: [0003] [0004] Among them, MTTF is the mean time between failures, and MTTR is the average time to repair. From the perspective of improving cost performance, it is more effective to improve MTTF in a fault-tolerant manner....

AI Technical Summary

Problems solved by technology

In the early storage solutions, the combination of Hadoop+Hbase was directly used to build indexes by combining row keys to improve retrieval performance. However, this method is only suitable for a small number of column clusters and has a single function. Once the index is established, it cannot be changed. The system function expansion brings inconvenience

There are also solutions that rely on the third-party paas (platform as a service) method. Although this method uses commercial platforms to simplify the technical problems of building high-performance clusters, since the data is not local, sensitive data cannot be well protected, and even more leaked risks of

Further, some researchers mainly use the integration of Hadoop platform and HIVE in the large-scale data storage method, which greatly reduces the query time of massive data, but because it is based on HDFS, once the data in Hive is imported, it cannot be changed. Regularly download the latest data

[0006] Meteorological data needs to be continuously collected and updated, and the data volume is large. Traditional storage methods occupy a large amount of space. With the increase in data volume, data maintenance costs increase sharply and retrieval time increases greatly, resulting in low data processing efficiency.

The traditional method uses a dedicated server to simply store meteorological data at a single point through a relational database (RDBMS), resulting in high costs, while using a large number of cheap machines on the hadoop platform to build a cluster will bring about the problem of distributed redundant storage of data

Images