Algorithm model for intelligently identifying quantitative relationship and anomaly thereof in log data

Abstract

The invention discloses an algorithm model for intelligently identifying a quantitative relationship and an anomaly thereof in log data. The algorithm model comprises training data preprocessing, log analysis, feature extraction, log quantitative relationship mining and anomaly detection. According to the method, an effective pruning method is provided, a complex multivariate log mode quantitative relation is converted into a binary log mode quantitative relation mining problem, and compared with an existing solution, the mining efficiency is greatly improved (from several hours to less than 10 minutes), and the algorithm effect is effectively improved. The problem of automatic division of the log fragments can be effectively solved by utilizing the fixed time window to divide the log fragments, and the window division in the time dimension is closer to the research that the log number corresponding to a log mode with a quantitative relationship in a short time meets the quantitative relationship of the log mode. Therefore, the windows in the time dimension instead of the windows in the number dimension are used.

Description

technical field [0001] The invention relates to the field of log anomaly detection, in particular to an algorithm model for intelligently identifying quantitative relationships and anomalies in log data. Background technique [0002] Log anomaly detection, that is, fault diagnosis based on log data, refers to analyzing log data generated during system operation by intelligent means to automatically discover system anomalies and diagnose system faults. Existing log anomaly detection systems are usually divided into three parts. The first part is log parsing, which converts unstructured logs into structured log patterns. Log text can be regarded as composed of constants and variables. Log parsing The purpose is to retain the constant part of the log text and replace the variable part with wildcards to generate a log pattern; the second part is feature extraction. After parsing the first part of the log, the original log text sequence can be converted into a log pattern sequenc...

AI Technical Summary

Problems solved by technology

[0004] (1) First, it is necessary to provide multiple log fragments to generate the log mode count matrix, and there is no effective automatic generation of a reasonable log fragment from the original large log data.

[0005] (2) Using the method of singular value decomposition, the method of counting the matrix from the extracted log pattern is too inefficient. When the number of log patterns or samples is large, the required calculation time is unacceptable; the second is to dig out Quantitative relationships are less precise

Images