Tuple tracking method and tuple tracking system for data flow processing

Abstract

The invention relates to a tuple tracking method and a tuple tracking system for data flow processing. The tuple tracking system comprises a tuple generator, a tupler tracker and a plurality of tuple processors. The tuple generator generates root tuples, processes the root tuples, and generates new tuples, and then transmits the new tuples to the different tuple processors. The tuple processors process the received tuples and generate new tuples. Each root tuple can generate a tuple tree after being processed, in the generation process of each tuple tree, the tuple generator transmits relevant information of the root tuple to the tuple tracker, then the tuple tracker can construct a tracking record of the root tuple, and the tuple tracker chooses one tuple tracking unit for each root tuple. When each tuple processor processes the tuple, relevant information of the tuple is sent to the tuple tracker to update identification positions of the tracking records. The tuple tracking method and the tuple tracking system for data flow processing can greatly reduce internal storage cost, achieve load balance of the tuple tracking units, and improve reliability of tuple processing.

Description

technical field [0001] The invention relates to the field of distributed data flow processing, in particular to a data flow processing-oriented tuple tracking method and system that ensures that each tuple that needs to be processed in the data flow will not be lost and cannot be processed. Background technique [0002] With the rise of technologies such as cloud computing and the Internet of Things, data is growing and accumulating at an unprecedented rate, and more and more appear in applications in the form of large-scale and continuous streams. The most typical application Monitoring applications, such as financial market monitoring, network monitoring, mobile object monitoring, intrusion detection, and ecosystem monitoring, etc., because these applications monitor real-time data, so the value of the data will continue to decrease over time , so low-latency processing is a key requirement for this type of application. For this reason, many data stream processing systems ...

AI Technical Summary

Problems solved by technology

The time overhead of upstream backup is less, only the upstream server needs to back up the output tuple in the output queue, but the recovery time of failure is also longer

[0007] However, the above backup technology is only applicable to the case where the failure granularity is the server. When the server does not fail, but the tuples processed by it are lost due to memory constraints and other reasons, the above technology cannot reprocess these tuples.

However, if the fault granularity is defined as each tuple in the process of data stream processing, then when the number of tuples is large, in order to ensure that each tuple is processed normally, the system needs to keep them in memory until they are processed normally After confirmation, so the memory overhead will be large

Images