Cloud computing based mass data processing system

A mass data processing and cloud computing technology, applied in the direction of electrical digital data processing, special data processing applications, computing, etc., can solve problems such as single point failure, difficulty in parallel processing, system function threats, etc., to achieve good load balance, reduce The effect of communication overhead

Inactive Publication Date: 2015-09-02
MASHANGYOU TECH CO LTD
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, we see several limitations of this approach, especially in distributed systems
[0003] First, the central database server is difficult to achieve load balancing of multiple nodes in the system
[0004] Second, it is easy to have a single point of failure, where a fault-tolerance problem could pose a threat to the functionality of the system
[0005] Third, it creates a very serious communication load, because the data distributed in each node must be delivered to the central server through the underlying network
Finally, this model is difficult to achieve parallel processing to take advantage of the computing advantages of the cloud platform architecture

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Cloud computing based mass data processing system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0017] In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. The following examples are only used to illustrate the technical solution of the present invention more clearly, but not to limit the protection scope of the present invention.

[0018] A specific embodiment of the present invention is,

[0019] Such as figure 1 As shown, a scalable distributed storage layer is provided, using the Hadoop system to maintain small distributed regional clusters, and then these clusters are treated as nodes in a larger shared-nothing cluster, managed by the Hadoop system. Each small cluster node is regarded as a slave node in the Hadoop system, and two master nodes are designated as coordinators of the Hadoop system. We call this design a distributed data warehouse using Hadoop. We store data in the distributed file system, Hadoop Distributed File Sys...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a cloud computing based mass data processing system which comprises a Hadoop system, a distributed regional small group, a master node and a distributed file system, wherein the distributed regional small group is viewed as a node in a bigger non-sharing cluster and is managed by the Hadoop system, the master node is a coordinator of the Hadoop system, and data are stored in the distributed file system. The cloud computing based mass data processing system provides excellent loading balance, has a fault tolerance function and meets the requirements on distributed and parallel processing and can greatly reduce communication expense.

Description

technical field [0001] The present invention relates to a data processing system, and more specifically, to a massive data processing system based on cloud computing. Background technique [0002] An important issue in the cloud computing architecture is how to design an efficient storage layer to handle massive data on the cloud computing platform. According to the design of Mayou Cloud Platform, data is naturally distributed and managed and stored, that is, all data is connected into a data group by a high-speed LAN. Massive data is generated through various applications on the cloud platform system. A possible data storage and query method is to use a centralized, relational database management system (DBMS) as the underlying data storage layer. However, we see several limitations of this approach, especially in distributed systems. [0003] First, it is difficult for the central database server to achieve load balancing among multiple nodes in the system. [0004] Sec...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06F9/50
CPCG06F9/5083G06F16/27
Inventor 陈勇胡中骥
Owner MASHANGYOU TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap