Method and device for realizing persistence in flow calculation application

Abstract

The invention discloses a method and a device for realizing persistence in flow calculation application. The method comprises the following steps that: when the current batch message is successfully consumed, whether the persistence operation needs to be performed or not is judged according to the first initial offset and the preset persistence interval; when the persistence operation needs to be performed, the persistence processing is carried out according to the message position indicated by the second initial offset; and after the persistence succeeds, the first initial offset and the second initial offset are updated into the initial offset of a next batch message. The persistence operation is performed after the persistence interval, and the disk persistence time interval is prolonged, so that the real-time calculation efficiency is greatly improved. During fault recovery, at most the batch message in the persistence interval needs to be consumed again; the performance bottleneck caused by frequent disk writing in the existing synchronous persistence process is avoided; the real-time calculation message throughput performance is improved by an order of magnitude; and meanwhile, the delay caused by the fault recovery is reduced to the second stage, and the real-time performance cannot be influenced.

Description

technical field [0001] The present application relates to stream computing technology, and more particularly, to a method and apparatus for realizing persistence in stream computing applications. Background technique [0002] Usually, in stream computing, a data stream is called a message, and a series of computations and processing on a data stream is called a consumption. [0003] Stream computing products are mainly used for real-time computing. Real-time calculations are usually performed in memory, and the results of calculations must be saved and displayed in some way. At present, the calculation results are mainly stored in two ways: caching or persisting to a disk such as a database (non-memory database). Among them, since the cache mode has no physical disk input / output (I / O), the cache mode has excellent message throughput; however, since the calculation results are not persistent, the cache mode has almost no fault tolerance, that is, once In the event of appli...

AI Technical Summary

Problems solved by technology

Among them, since the caching method has no physical disk input / output (I / O), the caching method has excellent message throughput; however, since the calculation results are not persisted, the caching method has almost no fault tolerance, that is, once In the event of application interruption, server downtime, cache emptying, etc., the calculation results stored in the cache will not be restored

The method of persisting to disk can achieve the highest level of fault tolerance. However, persisting to disk involves a large number of disk writes, which reduces the calculation speed of stream computing, and the execution efficiency is about lower than that of caching. One order of magnitude

Images