Log event extraction method and system based on log tree and parse tree

Abstract

The invention discloses a log event extraction method and system based on a log tree and a parse tree. The method is divided into two steps of preprocessing and log content parsing, and the method specifically comprises the steps of providing and maintaining a rule base composed of regular expressions and heuristic rules, and extracting a small part of logs to automatically generate a log format; recognizing the log as a log head and log content on line based on the log format; searching the analytic tree, and respectively calculating the similarity between the static field and the dynamic parameter in the log tree and the event tree by adopting the longest common substring and the longest common subvector; and matching the log tree and the event tree by adopting a clustering technology, and extracting events and corresponding parameters. In order to cope with the complexity of the log content, the preprocessing and log content analysis steps in the online event extraction method are improved. The workload of manually recognizing log formats is reduced, the problem that an existing method is difficult to identify events containing uncertain number of parameters is solved, and log events are extracted more accurately.

Description

technical field [0001] The invention belongs to the field of log analysis event extraction, in particular to a method and system for extracting log events based on log trees and parsing trees. Background technique [0002] With the rise of today's Internet technology, the scale of computing and communication infrastructure has expanded, and large-scale distributed systems have emerged. The log is the text data generated by the printout code embedded in the program, recording the current operating status and behavior mode of the system. In the process of system development and maintenance, domain experts realize real-time monitoring, anomaly detection, fault prediction and diagnosis of the system by analyzing logs. The scale of logs is expanding rapidly, and it is difficult to efficiently identify new abnormalities in the rapidly updated system and effectively eliminate system failures only by manually analyzing logs. Therefore, log analysis has gradually changed from offli...

AI Technical Summary

Problems solved by technology

However, existing event extraction methods formulate heuristic rules for global structural information such as the number of log fields or local content information such as log field categories, ignoring the hidden internal structural information between fields

After the log is matched to the log cluster, compare the fields of the log and the event, and replace the fields with different values ​​with wildcards, and extract the value as a parameter to update the event. This method also ignores the internal structure information of the original log, which can easily lead to static Fields are incorrectly extracted as parameters, event granularity is too coarse

Images