Low-coupling distributed streaming computation framework with master/slave architecture

Abstract

The embodiment of the invention provides a low-coupling distributed streaming computation framework with a master / slave architecture. The low-coupling distributed streaming computation framework comprises a distributed service environment, a cluster management master node, cluster work slave nodes and cluster service process nodes; the distributed service environment is used for storing the operation states, the load states and the task execution states of various distributed nodes; the cluster management master node is used for managing the operation states of cluster nodes and distributing tasks to the various cluster nodes; the cluster work slave nodes are used for executing tasks distributed by the cluster management master node and caching intermediate results of task execution in a Kafka; and the cluster service process nodes are used for consuming messages generated in the Kafka and obtaining service results. The low-coupling distributed streaming computation framework can be applied to streaming data process under multiple conditions, modules are fully decoupled, dynamic shrinkage and expansion can be achieved, redundant backup of data is achieved, and the service reliability can be achieved by means of the backup mechanism of the master node.

Description

technical field [0001] Embodiments of the present invention relate to the technical fields of big data processing and streaming computing, and in particular, to a low-coupling distributed streaming computing framework of a master / slave architecture. Background technique [0002] In recent years, with the rapid development of information technology, the amount of data has shown a trend of rapid growth. For massive data, the processing power of a single computer is far from enough, which promotes the research and development of distributed systems. The core idea of ​​a distributed computing system is "divide and conquer", which divides massive data sources into tasks, distributes the divided tasks to multiple computers for parallel processing, and merges the results of parallel processing into the final result. Distributed computer clusters are interconnected through the network, which can realize resource sharing, collaborative work, and parallel processing, and provide a uni...

AI Technical Summary

Problems solved by technology

Existing typical distributed stream computing frameworks include Storm, Sparkstreaming, Flink, etc. These frameworks have good real-time performance and fault tolerance in a distributed environment, but for specific business scenarios, the coupling is too high, which increases development and maintenance costs , reduce the heterogeneity between modules, and the code utilization rate of the framework is low, resulting in a certain waste of system resources

[0004] In the streaming computing scenario, the general streaming computing framework has the disadvantages of being relatively cumbersome, highly coupled, and low in heterogeneity

Images