Parallel data reflow method under stream computing environment

Abstract

The invention provides a parallel data reflow method oriented for real-time streaming computation. The method comprises the steps that step 1, initialization of three queues; step 2, initialization of a piping Data Queue; step 3, read requests are initiated by Spout of Topology to the Data Queue; step 4, data in the three queues is read by Data Queue; step 5, whether or not the queue pointed by ToP is empty is determined, if the queue is empty, step 6 is proceeded; if the queue is not empty, step 7 is proceeded; step 6, the data in the From queue is copied to the To queue, and the From queue is cleared; 7, data in the Data Queue is obtained by Topology, a Tuple is sent by current Task to downstream; step 8, the feedback of Tuple awaits for being sent by current Task, if the sending fails or times out and the feedback is not sent, the Tuple is opted to reflow; 9, whether or not the Topology can be stopped is determined, and if the Topology cannot be stopped, then step 4 is proceeded, otherwise, the steps are ended. By the parallel data reflow method oriented for real-time streaming computation, the data is stateless and has fault-tolerance, data computation latency is reduced, system response is increased, and the reflowed data is processed by priority at the first possible chance.

Description

technical field [0001] The invention relates to the field of information technology, in particular to a parallel data reflow method in a streaming computing environment. Background technique [0002] From social network information (to provide hot topics or real-time search) to advertising processing data engines, real-time stream computing is widely used in today's industry, such as Apache Storm, Twitter's Heron, Apache Flink, SparkStreaming, Samza, etc. In these systems, the generation of data is completely determined by the data source, and the dynamic change and inconsistency of the data source cause the rate of the data flow to present a bursty characteristic, and the bursty characteristic of the data flow often leads to the occurrence of overload , There are several reasons for overloading: network congestion, high resource utilization, interference, heterogeneity, IO high-frequency blocking, etc. Therefore, in real-time stream computing, overload is common and unavoi...

AI Technical Summary

Problems solved by technology

[0007] In order to overcome the deficiency that the stateful data processing method implemented in the existing real-time stream computing to solve data fault tolerance increases the complexity of the system. In addition, when the Topology is overloaded, Tuple shows frequent replay. This stateful Workarounds will undoubtedly increase the load on the system, and on large-scale distributed systems, any improvement in performance means a significant reduction in infrastructure costs, and a significant increase in end-user productivity

Images