Merging and storing method and system for massive data

Abstract

Disclosed is a mass data fusion storage method and system. The method comprises: a data fusion subsystem receives a write operation request sent by a structured storage engine or by an NoSQL storage engine, wherein the structured storage engine supports SQL structured data read and write operations, and the NoSQL storage engine supports key value NoSQL data read and write operations; the data fusion subsystem, in accordance with the write operation request, engages in a write operation in regard to a distributed file subsystem, wherein the distributed file subsystem is additionally used to receive a read operation request sent by a structured storage engine or by an NoSQL storage engine, and, in accordance with the read operation request, to execute the read operation. Achieved is a reduction in data redundancy and decreases system operation and maintenance costs.

Description

technical field [0001] The present invention relates to the communication field, in particular to a massive data fusion storage method and system. Background technique [0002] With the rapid development of the Internet, massive amounts of data are generated every day. The US Internet Data Center pointed out that the data on the Internet will increase by 50% every year, doubling every two years, and more than 90% of the data in the world is only generated in recent years. How to efficiently process these massive amounts of data has become a hot topic in recent years. [0003] In the existing mass data processing systems, there are structured storage engines that support SQL (Structured Query Language, Structured Query Language) for processing relational data, and NoSQL (Not Only SQL, unstructured query language) for processing non-relational data. query language) storage engine. Among them, the structured storage engine mainly completes the functions of structured data de...

AI Technical Summary

Problems solved by technology

For the same data, depending on the needs of the business, it may not only need to perform massive data query, statistics, grouping, and sorting operations through SQL, but also need to efficiently query and intelligently scan through key-value methods. It is difficult to implement in the system

It can be seen that the current massive data processing system mainly has the following problems: (1) The structured storage engine and the NoSQL storage engine are independent of each other in terms of physical data storage. Diverse business needs; (2) If a structured storage engine stores a copy of data to support SQL for business needs, a NoSQL storage engine also stores a copy of the same data to support key values. For massive data systems, there are a large number of Data redundancy, maintaining two copies of data at the same time will also increase the cost of operation and maintenance

[0005] For the problems that structured storage engines and NoSQL storage engines in related technologies cannot meet flexible and diverse business needs and increase operation and maintenance costs, no effective solutions have been proposed so far.

Images