System and method for stream processing

Abstract

A system and method for processing an input stream of events using a function f, to obtain an output stream of events are provided and are configured for: selecting from the input stream of events a plurality of consecutive events using a sliding window, as sliding window events; applying the function to the sliding window events, to obtain an output result value; iteratively: outputting the output result value on the output stream of events; splitting the sliding window events into a set of complying events that satisfy at least one filter test and a set of pending events; applying the function on the set of pending events to obtain at least one preliminary value; selecting from the stream of events a second plurality of events; adding the second plurality of events to the sliding windowevents; removing from the sliding window events the set of complying events, to obtain a new set of sliding window events; applying the function to the second plurality of events, and the at least onepreliminary value, to obtain a new output result value.

Description

Background technique [0001] In some of its embodiments, the present invention relates to a data stream processing system, and more specifically, it not only relates to distributed processing of data in a big data system. [0002] The term "big data" is used to refer to such a large and / or so complex data collection that traditional data processing applications cannot adequately handle such collections. One of the challenges of dealing with big data is analyzing the large amount of data in the collection. In some systems, data is an ordered sequence of data instances or events, called data streams or event streams. [0003] In a typical batch processing system, the data can be accessed as many times as needed to perform the required processing. In a stream processing system, data arrives continuously but cannot be stored for future reference. It may be necessary to constantly calculate mathematical or statistical analysis within the event stream at runtime. In some systems, large...

AI Technical Summary

Problems solved by technology

Additionally, the system may need to be scalable and have a fault-tolerant architecture

Images