Big-data parallel computing method and system based on distributed columnar storage

Abstract

The invention discloses a big-data parallel computing method and system based on distributed columnar storage. Data which is most often accessed currently is stored by using the NoSQL columnar storage based on a memory, the cache optimizing function is achieved, and quick data query is achieved; a distributed cluster architecture, big data storing demands are met, and the dynamic scalability of the data storage capacity is achieved; combined with a parallel computing framework based on Spark, the data analysis and the parallel operation of a business layer are achieved, and the computing speed is increased; the real-time data visual experience of the large-screen rolling analysis is achieved by using a graph and diagram engine. In the big-data parallel computing method and system, the memory processing performance and the parallel computing advantages of a distributed cloud server are given full play, the bottlenecks of a single server and serial computing performance are overcome, the redundant data transmission between data nodes is avoided, the real-time response speed of the system is increased, and quick big-data analysis is achieved.

Description

technical field [0001] The present invention relates to the technical field of big data processing, in particular to a large data parallel computing method and system based on distributed columnar storage. Background technique [0002] The rapid development of the Internet and the continuous upgrading and replacement of hardware have caused the data scale of various units such as governments and enterprises to show explosive growth, and gradually move towards massive data. Faced with the storage and processing requirements of massive data, traditional relational databases are mainly based on the operation of tables and data rows, which has gradually failed to meet user needs, and even restricts the storage and processing of massive data. Therefore, relying solely on traditional storage technology cannot meet the development and needs of the times. It is necessary to establish a new big data storage technology based on traditional processing technology to ensure that data sto...

AI Technical Summary

Problems solved by technology

[0005] In addition, traditional serial computing can no longer meet the needs of real-time query and analysis of big data, because the serial computing method requires tasks to be performed one by one in chronological order or priority processing order, which limits the existing CPU multi-core multi- The thread and distributed processing architecture can handle multi-task performance at the same time, but the calculation speed is slow

Images