Two-stage storage method combined with RDBMS (relational database management system) and Hadoop cloud storage

Abstract

The invention provides a two-stage storage method combined with an RDBMS (relational database management system) and Hadoop cloud storage. The two-stage storage method is characterized in that the front end adopts the RDBMS, and the back end adopts a Hadoop cloud end; the front end is used for collecting small files of structured data and storing in a database of the RDBMS; and when the capacity of the database achieves a certain quantity, the whole database can be used as a large file to be written into the Hadoop cloud end for storage, the database is simultaneously emptied, and the database can continuously accept the writing of the new data. According to the two-stage storage method provided by the invention, the lightweight relational database is utilized for meeting high-efficient writing and large-scale concurrent access requirements of the structured data, and the RDBMS can further automatically merge the mass small files into the large file so as to avoid the inconvenience caused by developing a tool for merging the small files; and the Hadoop cloud storage is utilized for realizing high-efficient storage of the mass files of the large database, so that the defects of Hadoop can be avoided, and the advantages of the cloud storage can be further fully utilized for realizing high-efficient storage and management. As long as the data meets the structuring requirements, the technical scheme provided by the invention can be used for high-efficient storage, and the two-stage storage method has certain universality.

Description

technical field [0001] The invention relates to the field of efficient storage and management of massive small files, in particular to an efficient storage method for massive structured data combined with RDBMS and Hadoop. Background technique [0002] Since the beginning of the 21st century, due to the explosive growth of information, there have been more and more application services for massive structured data. Typical applications include Weibo, instant messaging tool information, mobile phone text messages, and complex system logs. Such services often have the following characteristics: [0003] (1) The data has structured features, and its structure is relatively simple; [0004] (2) The data generation speed is very fast, generating tens of thousands or even hundreds of thousands of records per second; if stored in the form of traditional files, a large number of small files will be formed. [0005] (3) The data stock is very large, with more than tens of millions, ...

AI Technical Summary

Problems solved by technology

[0006] (4) Data is often "stored once and applied multiple times", and the real-time requirements for data reading and query are relatively high

Relational database (RDBMS) is one of the most effective means of managing structured data, but when the amount of data increases, the efficiency of the system will seriously drop, and it cannot provide safe and efficient storage management

Images