A real-time stream computing flow-aware elastic execution fault-tolerant system

Abstract

The invention discloses a real-time stream computing flow rate perception elastic execution fault-tolerant system, which transforms the tight coupling relationship between tuple emission, calculation, state maintenance and processing units in the traditional stream computing framework into a modular and professional loose coupling relationship, Combined with modules such as flow rate perception, elastic execution, and state management, the system can finely control the processing time of each stage to ensure the delay of the entire application. Flow rate awareness enables the system to respond quickly to changes in incoming flow or changes in available resources. The distribution module can flexibly change the distribution relationship and send tuples in batches in a timely manner, further reducing resource consumption and sending delay. Elastic execution enables the system to transparently adjust the parallelism of processing units and nodes to users, so as to maintain application stability and guarantee latency. State management and persistent storage make the system incredibly fault-tolerant, saving an appropriate number of tuples speeds up application recovery, and combined with elastic execution of parallel recovery further improves failure recovery speed.

Description

technical field [0001] The invention belongs to the field of big data distributed computing, and more specifically relates to a real-time stream computing flow rate-aware elastic execution fault-tolerant system. Background technique [0002] Currently, real-time stream computing has received extensive attention. Academia and industry have developed multiple stream computing systems: Storm, S4, D-Streams, Puma, Flume, Streambase, Timestream, System S, etc. Developers use the API provided by the above system to describe the processing logic to build applications. Since the amount of data processed by such applications usually exceeds the processing capacity of a single machine and requires multiple stages of calculation and processing, the system deploys the application to multiple machines for distributed execution. Streaming applications have the following typical characteristics: long-running, real-time data flow into the system, unknowable inflow rate, error accumulation...

AI Technical Summary

Problems solved by technology

[0007] However, the above method has the following disadvantages: it cannot make full use of the resources of the entire cluster for load balancing and delay guarantee, lacks a scheduling strategy for coordinating the processing speed of nodes and available resources of nodes; it cannot flexibly and quickly generate an overall strategy to deal with the cluster perspective according to the change of flow rate ( Such as adjusting the number of operators and increasing resource usage); when a node fails, it takes a lot of time to retrieve the node state and replay and reprocess the tuple after the timestamp of the detection point from the original stage

Images