Stream computing system and log data processing method of stream computing system

Abstract

The embodiment of the present application discloses a stream computing system and a log processing method thereof. The streaming computing system includes: a log collection module, which collects log data from the log output terminal and sends it to the message system; a message system, which stores the log data in the message queue; a status monitoring module, which monitors the status of the log output terminal and reports to the progress monitoring module; the log processing module obtains the log data from the message system and parses the log output end identification of the output log data and the log data output time associated with the log output end identification, and combines the log output end identification and The associated log data output time is reported to the progress monitoring module; the progress monitoring module determines the unprocessed logs in the flow computing system according to the identification of the log output end, the associated log data output time, and the status of the log output end The earliest output time of the data. The streaming computing system realizes the automatic advancement of the global water level.

Description

technical field [0001] The embodiments of the present application relate to the field of computer technology, specifically to the field of data processing technology, and in particular to a streaming computing system and a log data processing method thereof. Background technique [0002] In the streaming computing scenario, the water level is used to mark the earliest timestamp of unfinished data in the system. Water level dumping refers to dumping the data from the streaming system into the folder corresponding to the window according to the window size (for example, 5 minutes), and marking the end of the window after the window ends, and the data in the file is complete. After the window ends, downstream consumers can further process the data in the window. [0003] However, due to the complexity of the system environment, the state of some log output terminals may be abnormal, and the network may also fluctuate, making it impossible to accurately place the log data into ...

AI Technical Summary

Problems solved by technology

[0003] However, due to the complexity of the system environment, the state of some log output terminals may be abnormal, and the network may also fluctuate, making it impossible to accurately place the log data into the corresponding window. As a result, it is difficult to accurately measure the global water level of the system. Difficult to advance

Images