Support high concurrency distributed memory computing cluster system

Abstract

The application of the present invention provides a distributed memory computing cluster system that maintains high concurrency, including: a Master node, multiple Node nodes, cluster management components and a database, and the Master node. The main functions of the Master node are receiving requests, pre-analyzing requests, allocating and scheduling Jobs, cluster monitoring, collecting and returning analysis results. Master node includes request analyzer, job scheduling controller, job monitor, cluster monitor, request queue, job queue and time-consuming job queue, Node node includes task analyzer, task scheduling controller, task monitor, data loading engine , task queue, data buffer and intermediate result buffer, the cluster management component is mainly used to monitor and maintain the health status of the entire cluster, and the database is used to store historical fragmented data and real-time data. It can effectively reduce "framework consumption" and realize data preloading, request preanalysis and pure memory operation.

Description

technical field [0001] The invention relates to the fields of software engineering, big data, distributed storage and computing, and in particular to a distributed memory computing cluster system supporting high concurrency. Background technique [0002] With the development of computer hardware and big data technology, in this era of information explosion, people are eager to quickly count and obtain valuable information from massive data, which has important decision-making and reference for the development of enterprises and people's lives effect. [0003] The current feasible technical solutions are summarized as follows: use hadoop or spark technology, for example, use hdfs / habse as the data storage engine, use map / reduce or spark technology as the offline data analysis engine, use kafka, spark-streaming or storm as the real-time data analysis engine. [0004] Although the above technical solutions can also analyze and calculate the data, I personally think that there...

AI Technical Summary

Problems solved by technology

Although the entire big data world was once sensationalized by the replacement of MAP / REDUCE by SPARK memory computing, SPARK is not a pure memory operation, and many intermediate results still have to be saved in the form of files, especially when shuffle occurs, that is, data needs to be transmitted across the network Sometimes, the intermediate result data is generally saved as a file more or less due to insufficient memory, and some even need to be compressed and saved, which reduces the performance of the system to a certain extent

Images