Cloud computing based mass data processing system

Abstract

The invention discloses a cloud computing based mass data processing system which comprises a Hadoop system, a distributed regional small group, a master node and a distributed file system, wherein the distributed regional small group is viewed as a node in a bigger non-sharing cluster and is managed by the Hadoop system, the master node is a coordinator of the Hadoop system, and data are stored in the distributed file system. The cloud computing based mass data processing system provides excellent loading balance, has a fault tolerance function and meets the requirements on distributed and parallel processing and can greatly reduce communication expense.

Description

technical field

[0001] The present invention relates to a data processing system, and more specifically, to a massive data processing system based on cloud computing. Background technique

[0002] An important issue in the cloud computing architecture is how to design an efficient storage layer to handle massive data on the cloud computing platform. According to the design of Mayou Cloud Platform, data is naturally distributed and managed and stored, that is, all data is connected into a data group by a high-speed LAN. Massive data is generated through various applications on the cloud platform system. A possible data storage and query method is to use a centralized, relational database management system (DBMS) as the underlying data storage layer. However, we see several limitations of this approach, especially in distributed systems.

[0003] First, it is difficult for the central database server to achieve load balancing among multiple nodes in the system.

[0004] Sec...

Images