Heartbeat propagation in a distributed stream processing system

Abstract

An event processing system for distributing a continuous query, and processing events in an event stream, is disclosed. The system can synchronize timestamps across all partitions by adding a heartbeat propagator component propagating a heartbeat signal for silent partitions. The system can further include a timestamp tracker component that tracks the maximum observed timestamp across all partitions for each batch, and sends this to each partition. The system can distribute partitions of the events to the worker nodes, and instruct a heartbeat propagator component to generate a heartbeat for a first worker node corresponding to an event in a different partition. The system receives outputs from the worker nodes and combines these into a single result corresponding to a continuous query. The system can further implement a timestamp tracker to select a highest timestamp value of a previous batch, and send the value to the worker nodes.

Description

CROSS-REFERENCE TO RELATED APPLICATION[0001]The present application is a non-provisional of India Application No. 201741034654, filed Sep. 29, 2017, the entire contents of which is herein incorporated by reference for all purposes. This application is also related to application Ser. No. 15 / 696,501, filed Sep. 6, 2017; entitled “PROCESSING TIMESTAMPS AND HEARTBEAT EVENTS FOR AUTOMATIC TIME PROGRESSION;” application Ser. No. 15 / 700,784, filed Sep. 11, 2017 entitled “DATA SERIALIZATION IN A DISTRIBUTED EVENT PROCESSING SYSTEM;” application Ser. No. 15 / 700,862, filed Sep. 11, 2017, entitled “GRAPH GENERATION FOR A DISTRIBUTED EVENT PROCESSING SYSTEM;” application Ser. No. 15 / 700,914, filed Sep. 11, 2017, entitled “CLUSTERING EVENT PROCESSING ENGINES;” and application Ser. No. 15 / 701,019, filed Sep. 11, 2017, entitled “DATA PARTITIONING AND PARALLELISM IN A DISTRIBUTED EVENT PROCESSING SYSTEM,” the entire contents of which are hereby incorporated by reference as if fully set forth herei...

AI Technical Summary

Benefits of technology

The patent describes techniques for distributing a continuous query and processing events of an event stream. The system receives a plurality of events in a stream and distributes a query plan to a master node and a plurality of worker nodes. Each worker node implements a heartbeat propagator component and a continuous query engine. The master node instructs a heartbeat propagator component on a worker node to propagate a heartbeat for the worker node. The worker node receives the heartbeat and the query plan and combines the event results into a single result for the continuous query. The system also includes a timestamp tracker to track highest partition timestamp values and a load balancing function to optimize distribution of the query plan. The techniques described in the patent facilitate efficient processing of continuous queries and event streams.

Problems solved by technology

While some vendors now offer product suites geared towards event streams processing, these product offerings still lack the processing flexibility required for handling today's event processing needs.

Images