A log real-time analysis method and system

Abstract

The invention discloses a real-time log analysis method and system, and relates to the field of data processing. The method comprises the steps of judging a current working node as a registered node by a log parsing client and obtaining a to-be-analyzed log directory of the current working node; judging that an updated log file exists in the log directory, starting a thread pool and updating parsing of the log file, carrying out vertical data splitting according to a system flag and then storing the data into different data tables; and finally storing the data tables into a log database. The system comprises a log recording module and a log analysis module. The storage capacity of the log is reduced under the premise of not affecting a real-time monitoring function; the elapsed time in the change from execution of the system method to reflection of the execution result from an interface is smaller than one minute; a monitoring system can be horizontally expanded; and performance extension can be finished only by simple deployment.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a log real-time analysis method and system. Background technique [0002] At present, log monitoring is mostly implemented based on results. It is used to detect whether the current function is still available, and find out that it is not available as soon as possible. However, the operation has been implemented and the result is obtained. Correcting it is equivalent to starting again. Therefore, the existing monitoring method belongs to The monitoring method of repairing the past. However, how to discover the tendency of the monitored system to deteriorate before the problem occurs is a new problem. At the same time, it is also necessary to ensure that the granularity of monitoring can support the execution method of the core functions of the system. [0003] Most of the existing log-based monitoring solutions are performed by running the system once and recording it once, then r...

AI Technical Summary

Problems solved by technology

[0004] 1. The amount of logs is too large. For those systems that are accessed frequently, the amount of logs generated every day will reach G level;

[0005] 2. Due to the large amount of logs, the statistical analysis takes a long time;

[0006] 3. Most of the data is stored in a centralized manner. As the number of monitored systems increases, the aggregated logs will become larger and larger, resulting in a slow analysis process and a sharp decline in database query performance, which cannot meet the "real-time" requirements. Expansion Need to develop

Images