Log template rapid extraction method and system based on association analysis and time window

Abstract

The invention belongs to the technical field of system log analysis, and discloses a log template rapid extraction method and system based on correlation analysis and a time window. The method comprises the steps of log hierarchical relationship extraction of extracting a plurality of seed types from a system log with correlation, wherein the seed types serve as all log models of events of the type; and constructing a template tree, wherein the processed field logs with the same field are used as father nodes in the tree structure. By respectively extracting the data set templates from a manual clustering result and three template extraction methods by using a Rand _ index method, and respectively comparing the similarities of the data set templates, in the three compared template extraction methods, an online template extraction model has the lowest data set template extraction efficiency, and the correction rate is about 50%, a template extraction model has the highest efficiency, and the correction rate is about 90%, so that the template extraction model based on the association relationship provided by the invention is more reliable.

Description

technical field [0001] The invention belongs to the technical field of system log analysis, in particular to a method and system for quickly extracting log templates based on correlation analysis and time windows. Background technique [0002] Currently, the closest prior art: [0003] The distributed characteristics of the Internet of Things system and the complexity of its services make the logs related to system interaction an important basis for system managers to monitor the system status. System logs can be used to discover system security problems and trace the root cause of problems based on system logs, but it is difficult to directly extract network events from logs. Usually, a network management system (NMS) can be used to monitor the network, and a network management protocol (SNMP) can be used with triggers to detect the network and list relevant log information. [0004] However, different events may have the same impact on the network load, and specific even...

AI Technical Summary

Problems solved by technology

[0008] (2) The level used to identify log events cannot accurately indicate the type of event, nor can it be directly used for anomaly detection

[0009] (3) The content of the log is complex, and various devices in the system and the network will also generate logs. Prior knowledge is required to find the accurate meaning in the log

[0011] The real-time requirements for system health management are high, and the existing technology is difficult to extract and analyze log network events in the scenario of massive logs

For real-time changes in the log, if the model parameters are updated in full, it will consume a lot of memory and CPU

Therefore, it is very difficult to analyze a large number of dynamically changing logs.

Images