Real-time streaming computing flow speed perceiving elastic execution tolerant system

Abstract

The invention discloses a real-time streaming computing flow speed perceiving elastic execution tolerant system. According to the system, a tight coupling relation of tuple transmitting, computing, state maintaining and processing units in a traditional streaming computing frame is changed into a modular and specialized loose coupling relation, in combination with a flow speed perceiving module, an elastic execution module, a state management module and the like, the system can control the processing time of all stages precisely, and thus display of a whole application is guaranteed. The system can conduct quick response to input steam changes or available resource changes quickly through flow speed perceiving. A distribution module can change the distribution relation flexibly and send tuples in batches timely, and resource consumption and sending delay are further reduced. The system can adjust the parallel numbers of processing units and nodes transparently through elastic execution so as to keep the application stable and guarantee delay. The system is made to have the unbelievable tolerant capacity through state management and persistent storage. The proper numbers of tuples are stored, the application recovery is accelerated, and the failure recovery speed is further increased in combination with parallel recovery of elastic execution.

Description

technical field [0001] The invention belongs to the field of big data distributed computing, and more specifically relates to a real-time stream computing flow rate-aware elastic execution fault-tolerant system. Background technique [0002] Currently, real-time stream computing has received extensive attention. Academia and industry have developed multiple stream computing systems: Storm, S4, D-Streams, Puma, Flume, Streambase, Timestream, System S, etc. Developers use the API provided by the above system to describe the processing logic to build applications. Since the amount of data processed by such applications usually exceeds the processing capacity of a single machine and requires multiple stages of calculation and processing, the system deploys the application to multiple machines for distributed execution. Streaming applications have the following typical characteristics: long-running, real-time data flow into the system, unknowable inflow rate, error accumulation...

AI Technical Summary

Problems solved by technology

[0007] However, the above method has the following disadvantages: it cannot make full use of the resources of the entire cluster for load balancing and delay guarantee, lacks a scheduling strategy for coordinating the processing speed of nodes and available resources of nodes; it cannot flexibly and quickly generate an overall strategy to deal with the cluster perspective according to the change of flow rate ( Such as adjusting the number of operators and increasing resource usage); when a node fails, it takes a lot of time to retrieve the node state and replay and reprocess the tuple after the timestamp of the detection point from the original stage

Images