Data parallel processing system based on Cassandra

Abstract

The invention discloses a data parallel processing system based on Cassandra. The data parallel processing system based on the Cassandra comprises a Hadoop main node, a plurality of Hadoop auxiliary nodes and a Cassandra storage end arranged on the Hadoop auxiliary node, wherein the main node comprises a user interface module, a Cassandra inquiring module, a job scheduling module, a job queue module, and a job tracker, wherein the auxiliary node comprises a task tracker, an input module, an output module and a Mapreduce module, the user interface module is used for receiving a user request, and judging that the type of the user request is a data inquiring request, or a submitting data processing job request, or a job information inquiring request, if the type of the user request is the data inquiring request, the user interface module sends the data inquiring request to the Cassandra inquiring module, if the type of the user request is the submitting data processing job request or the job information inquiring request, and the user interface module sends the submitting data processing job request or the job information inquiring request to the job scheduling module. The data parallel processing system based on the Cassandra has the advantages of being high in reliability, good in expansibility, and high in a throughput rate. The data parallel processing system based on the Cassandra has the capacity of simply inquiring and rapidly responding to the data, and meanwhile has the complex processing capacity to mass data.

Description

technical field [0001] The invention belongs to the field of distributed computing and system structure in the computer field, and more specifically relates to a Cassandra-based data parallel processing system. Background technique [0002] Cassandra is an open source, distributed, centerless, elastically scalable, highly available, fault-tolerant, adjustable consistency, column-oriented non-relational database. It is based on the distributed design of Amazon's Dynamo database and the data model of Google's BigTable, created by Facebook, which has already achieved applications in some of the most popular websites. At present, with the rise of Web2.0, the amount of data is increasing rapidly, and the storage and processing requirements of massive data pose a challenge to traditional relational databases, because traditional relational databases cannot meet the needs of ultra-large-scale and high-concurrency data. Process demand. For example, a Web2.0 website needs to genera...

AI Technical Summary

Problems solved by technology

[0004] For the defects of the prior art, the purpose of the present invention is to provide a data parallel processing system based on Cassandra, aiming to solve the deficiency of the existing Cassandra system for complex data processing functions, the system has high reliability, It has the advantages of good scalability, high throughput rate and the ability to quickly respond to simple data queries, and at the same time has the ability to complexly process massive amounts of data

Images