Message reliable processing guarantee method of real-time flow calculating frame based on Storm

Abstract

The invention discloses a message reliable processing guarantee method of a real-time flow calculating frame based on Storm. The method comprises the steps: 1, in a preprocessing phase, initialization work is carried out on an environment; 2, data which have already been sent and are in a calculating state are tracked in a cluster calculation process; 3, when an emission task monitors a signal that the message processing is successful, tracking information of all sub-tuples which belong to the task in a cache region are emptied; 4, when the emission task monitors a signal that the message processing is unsuccessful, the position of the task producing the processing failure and data to be recovered are located; 5, a message recovery program is constructed according to the tracking information and xml files, the data to be recovered are read from the cache region, and the message recovery program is executed; 6, the cache region is emptied, and a memory space is released. According to the method, when a message is recovered, large-scale repetitive computation existing in a complex topology business is avoided, the calculated amount of message recovery is effectively reduced, the performance of processing data under the whole business scene is promoted, and the low delay requirement of real-time processing can be ensured.

Description

technical field [0001] The invention belongs to the fields of massive data processing, real-time stream computing and fault tolerance, and more specifically relates to a message reliable processing guarantee method based on the Storm real-time stream computing framework. Background technique [0002] In recent years, the demand for big data processing has been increasing, and how to deal with huge amounts of data is full of challenges. With the further development of the Internet, from the browsing type of the portal website to the search type to the interactive transmission type of SNS relationship, and e-commerce will make the circulation link in life online. The requirement for efficiency makes people's requirements for real-time performance further improved, and the interaction of information is developing in the direction of the information network, which will inevitably bring about the cross-correlation of various dimensions of data, and the data explosion is inevitabl...

AI Technical Summary

Problems solved by technology

[0007] Aiming at the above defects or improvement needs of the prior art, the present invention provides a method for guaranteeing reliable processing of messages based on the Storm real-time stream computing framework. In the case of high computational logic complexity of the processing task, once a sub-tuple fails to be processed, the technical problem of message recovery is very expensive

Images