A low-coupling distributed streaming computing system with master/slave architecture

Abstract

The embodiment of the present invention provides a low-coupling distributed stream computing system of a master / slave architecture, including: a distributed service environment, a cluster management master node, a cluster working slave node, and a cluster business processing node; The service environment is used to store the running state, load state and task execution state of each distributed node; the cluster management master node is used to manage the running state of the cluster nodes and distribute tasks to each cluster node; the work of the cluster is from Nodes are used to execute tasks distributed by the cluster management master node, and cache intermediate results of task execution in Kafka; business processing nodes of the cluster are used to consume messages generated in Kafka and obtain business results. It can be applied to streaming data processing in a variety of situations, fully decoupled between modules, can realize dynamic shrinkage and expansion, and realize redundant backup of data. The backup mechanism of the master node can realize service reliability.

Description

technical field [0001] Embodiments of the present invention relate to the technical fields of big data processing and streaming computing, and in particular, to a low-coupling distributed streaming computing framework of a master / slave architecture. Background technique [0002] In recent years, with the rapid development of information technology, the amount of data has shown a trend of rapid growth. For massive data, the processing power of a single computer is far from enough, which promotes the research and development of distributed systems. The core idea of ​​a distributed computing system is "divide and conquer", which divides massive data sources into tasks, distributes the divided tasks to multiple computers for parallel processing, and merges the results of parallel processing into the final result. Distributed computer clusters are interconnected through the network, which can realize resource sharing, collaborative work, and parallel processing, and provide a uni...

AI Technical Summary

Problems solved by technology

Existing typical distributed stream computing frameworks include Storm, Sparkstreaming, Flink, etc. These frameworks have good real-time performance and fault tolerance in a distributed environment, but for specific business scenarios, the coupling is too high, which increases development and maintenance costs , reduce the heterogeneity between modules, and the code utilization rate of the framework is low, resulting in a certain waste of system resources

[0004] In the streaming computing scenario, the general streaming computing framework has the disadvantages of being relatively cumbersome, highly coupled, and low in heterogeneity

Images