Support high concurrency distributed memory computing cluster system

Abstract

The application of the invention provides a distributed memory computing cluster system with high concurrency, comprising a Master node, a plurality of Node nodes, a cluster management unit and a database, and a Master node. The main functions of the Master node are to receive a request, pre-analyze a request, allocate and dispatch a job, monitor a cluster, collect and return an analysis result. The Master node includes a request analyzer, a job scheduling controller, a Job Monitor, a Cluster Monitor, a Request queue, Job queues and time-consuming job queues, the node includes a task analyzer,a task scheduling controller, a task monitor, a data loading engine, a task queue, a data buffer and an intermediate result buffer. A cluster management module is mainly used to monitor and maintainthe health status of the whole cluster, database, for storing historical fragmentation data and real-time data. The node is composed of a task analyzer, a task scheduling controller, a task monitor, adata loading engine, a task queue, a data buffer and an intermediate result buffer. It can effectively reduce the 'framework consumption', and can achieve data preload, request pre-analysis and purememory operation.

Description

technical field [0001] The invention relates to the fields of software engineering, big data, distributed storage and computing, and in particular to a distributed memory computing cluster system supporting high concurrency. Background technique [0002] With the development of computer hardware and big data technology, in this era of information explosion, people are eager to quickly count and obtain valuable information from massive data, which has important decision-making and reference for the development of enterprises and people's lives effect. [0003] The current feasible technical solutions are summarized as follows: use hadoop or spark technology, for example, use hdfs / habse as the data storage engine, use map / reduce or spark technology as the offline data analysis engine, use kafka, spark-streaming or storm as the real-time data analysis engine. [0004] Although the above technical solutions can also analyze and calculate the data, I personally think that there...

AI Technical Summary

Problems solved by technology

Although the entire big data world was once sensationalized by the replacement of MAP / REDUCE by SPARK memory computing, SPARK is not a pure memory operation, and many intermediate results still have to be saved in the form of files, especially when shuffle occurs, that is, data needs to be transmitted across the network Sometimes, the intermediate result data is generally saved as a file more or less due to insufficient memory, and some even need to be compressed and saved, which reduces the performance of the system to a certain extent

Images