Storage system of distributed real-time database

Abstract

The present invention provides a storage system of a distributed real-time database. The system comprises: a distributed data gateway module, used for data reception and querying agent; a distributed message queue module, used for implementing data release and subscription by using a distributed message queue; a distributed real-time streaming computing service module, used for realizing a message subscription service, a memory snapshot storage service and a persistent storage service based on a storm real-time streaming computing framework; a distributed cache service module, used for storing a data snapshot by using a distributed cache; and a distributed storage service module, used for performing distributed storage service by using a Nosql database HBase, and implementing data retrieval by using a distributed search engine Solr. The storage system provided by the present invention effectively increases the level of intelligence and informatization of enterprises, and mines potential data value by using big data technology, and provides a solid data basis for transformation and development of the enterprises.

Description

Technical field [0001] The present invention relates to a distributed real-time database, in particular to the architecture of a storage system of the distributed real-time database. The invention can be widely used in industrial big data storage. Background technique [0002] With the continuous advancement of Industry 4.0, the intelligentization of industrial equipment and the information transformation of enterprises will advance rapidly, which will also bring about an explosive growth of data. The traditional real-time database is in terms of point scale, data distribution, reliability, and Many aspects such as scalability present huge challenges. A comparative study of PI and PHD, the world’s leading real-time database products, found that these products have the following limitations when facing industrial big data: [0003] (1) The data storage service fails to implement a distributed architecture. When the stand-alone service is abnormal, the service will be unavailable, ...

AI Technical Summary

Problems solved by technology

[0003] (1) The data storage service fails to implement a distributed architecture. When the stand-alone service is abnormal, the service will be unavailable and cannot provide retrieval and storage functions;

[0004] (2) There is no copy mechanism for data storage, and the security of data needs to be backed up by users themselves, so there is no guarantee that all data will have a reliable backup when the disk is damaged;

[0005] (3) Lack of flexible dynamic expansion capability, when the storage performance or storage capacity reaches the bottleneck, it is difficult to achieve dynamic horizontal expansion;

[0006] (4) The data retrieval capability is limited, and only supports retrieval conditions based on timestamp, and does not support retrieval by value range or fuzzy;

[0007] (5) The computing power is limited, and it can only be calculated according to the limited calculation method provided by it, and it cannot effectively use the big data distributed computing technology to realize the calculation and analysis of massive data

[0020] Comparison of technical points: This technical paper theoretically expounds the high-performance data storage and retrieval mechanism of distributed real-time databases and gives the corresponding algorithms, but fails to specifically give an effective and scalable architecture design, and the entire architecture design is too general , for the reliability guarantee of data transmission, real-time stream data processing and distributed index mechanism, and the support for future distributed computing, no clear design plan has been given.

Images