Big data multi-level storage architecture

Abstract

The invention belongs to the technical field of big data storage, and particularly relates to a big data multi-level storage architecture which comprises three levels, namely a first level, a second level and a third level, wherein the hierarchy I comprises the following steps: only storing hot spot data; an Hbase is used for providing low-delay random reading and writing; an SSD hard disk is usedas hardware to provide good performance; the hierarchy II comprises the following steps: storing relative popular data; spark is used for providing efficient large-scale analysis and calculation; themethod comprises the following steps of: storing data by using an open-source Apache Hadoop distributed file system; and a Parquet format is used for data storage, so that the Spark calculation efficiency is improved. According to the method, a multi-level big data architecture platform is built by utilizing different big data frameworks and technologies, and different technologies and solutionsare applied to different use scenes, so that the service performance is improved, and the calculation time and the hardware cost are reduced.

Description

technical field [0001] The invention relates to the technical field of big data storage, in particular to a big data multi-level storage architecture. Background technique [0002] With the continuous development of business and time, and the continuous expansion of user and product-related data, the traditional single-database or single-model storage method can no longer meet the growing demand for big data. It is necessary to read out the personal data related to a specific user from a large amount of data, and to perform batch analysis operations on a large range of user data over a long period of time, etc. [0003] In addition, when the data volume of a single table in the previous traditional relational database continues to increase, it will have a great negative impact on the performance of reading and writing, which will eventually lead to the unavailability of the service. However, modern big data tools also have their own limitations in usage scenarios. For examp...

AI Technical Summary

Problems solved by technology

[0002] With the continuous development of business and time, and the continuous expansion of user and product-related data, the traditional single-database or single-model storage method can no longer meet the growing demand for big data. Read out the personal data related to a specific user from a large amount of data, and perform batch analysis operations on a large range of user data over a long period of time, etc.

[0003] In addition, when the data volume of a single table in the previous traditional relational database continues to increase, it will have a great negative impact on the performance of reading and writing, which will eventually lead to the unavailability of the service.

However, modern big data tools also have their own limitations in usage scenarios. For example, storage suitable for random reading and writing is not suitable for batch and large-scale reading and writing, and vice versa.

At the same time, due to the increase in the amount of data, the subsequent increase in the cost of storage devices is also a problem that cannot be ignored at present.

Images