Mass information storage system and implementation method

Abstract

The invention provides a mass information storage system. The system comprises a storage system, a metadata server, a coordinator and a data distribution and balance module, wherein the storage system has a data storage function; the metadata server is used for receiving and saving metadata from the storage system; the coordinator is used for receiving data which needs to be written by a client into the storage system, organizing the received data to a key value form and sending to the data distribution and balance module, and the coordinator is further used for receiving an inquiry command of the client, instructing a node of the server to implement an inquiry operation, receiving an inquiry result and returning to the client; and the data distribution and balance module is used for distributing the data to all the nodes of the server of the storage system by applying a consistent hashing algorithm.

Description

technical field [0001] The invention relates to the technical field of data storage, in particular to a massive information storage system and an implementation method. Background technique [0002] With the rapid development and wide application of Internet technology, the number of Internet users and the amount of data are showing explosive growth. Various types of applications emerge in an endless stream, and traditional relational databases show certain limitations in terms of high concurrent access, massive data processing, and large-scale deployment. [0003] In the application scenarios involved in the smart grid, the amount of data collected regularly is very large, and the throughput of different relational databases is difficult to meet the requirements. [0004] The NOSQL database is produced along with the vigorous development of cloud computing technology. It has the characteristics of low-latency read and write speed, support for massive data, large-scale clus...

AI Technical Summary

Problems solved by technology

However, this kind of query and analysis needs to be converted into a MapReduce program to run, the speed is relatively slow, and it is difficult to provide online complex query functions

[0007] In addition, in this field, the MySQL database is also a very commonly used relational database. Although the MySQL database can implement complex query functions, the disadvantage lies in the limited read and write performance of a single MySQL database, which makes it difficult to withstand the regular flooding in power grid application scenarios. Massive data, the throughput is difficult to meet the requirements, if the data cannot be persisted within a certain period of time, there may be a danger of being overwritten by the next wave of data, resulting in data loss, and it is also difficult to handle a large number of access and query requests

Images