Log mode extraction and matching method

Abstract

The invention relates to a log mode extraction and matching method, which comprises the following steps of S1, cleaning historical log data to obtain cleaned historical log data; S2, preprocessing thecleaned historical log data to obtain preprocessed historical log data; S3, obtaining a historical word vector based on the preprocessed historical log data; S4, inputting the historical word vectorsinto a twin LSTM network, and extracting a log mode; and S5, based on the to-be-matched log data, the log mode and the twin LSTM network, performing log mode matching of the to-be-matched log data. Compared with the prior art, the data volume of log analysis is reduced, the log pattern extraction efficiency is improved, the log pattern matching efficiency is improved, and the reliability of a logpattern extraction result and the reliability of a log pattern matching result are effectively improved.

Description

technical field

[0001] The invention relates to the field of data mining, in particular to a log pattern extraction and matching method. Background technique

[0002] Nowadays, the scale and complexity of information systems are increasing significantly. The challenges faced by system operation and maintenance are becoming increasingly severe, and the required labor and material costs are constantly increasing. In severe cases, they may even exceed the construction cost of the system itself. The operation and maintenance of the system is mainly through continuous monitoring and analysis of system logs, performance counters and application resource usage patterns, assessing system health and determining performance changes. It mainly relies on various types of running status record data generated by the system. The common ones are performance index record data generated regularly by performance counters, such as KPI data, and system logs generated in real time, such as event ...

Examples

Embodiment

[0036] This embodiment provides a log pattern extraction and matching method, such as figure 1 shown, including the following steps:

[0037] Step S1: Cleaning the historical log data to obtain the cleaned historical log data;

[0038] Step S2: Preprocessing the cleaned historical log data to obtain the preprocessed historical log data;

[0039] Step S3: Obtain historical word vectors based on the preprocessed historical log data;

[0040] Step S4: Input the historical word vector into the Siamese LSTM network to extract the log pattern;

[0041] The matching of the log pattern is based on the log pattern obtained by the log pattern extraction method and the twin LSTM network, and the log pattern matching of the log data to be matched is performed.

[0042] in particular:

[0043] 1) Extract log mode

[0044] Such as image 3 As shown, for the historical log data of all components that appear in the system, first scan them one by one, perform simple data cleaning, and th...

technical field

[0001] The invention relates to the field of data mining, in particular to a log pattern extraction and matching method. Background technique

[0002] Nowadays, the scale and complexity of information systems are increasing significantly. The challenges faced by system operation and maintenance are becoming increasingly severe, and the required labor and material costs are constantly increasing. In severe cases, they may even exceed the construction cost of the system itself. The operation and maintenance of the system is mainly through continuous monitoring and analysis of system logs, performance counters and application resource usage patterns, assessing system health and determining performance changes. It mainly relies on various types of running status record data generated by the system. The common ones are performance index record data generated regularly by performance counters, such as KPI data, and system logs generated in real time, such as event ...

Examples

Embodiment

[0036] This embodiment provides a log pattern extraction and matching method, such as figure 1 shown, including the following steps:

[0037] Step S1: Cleaning the historical log data to obtain the cleaned historical log data;

[0038] Step S2: Preprocessing the cleaned historical log data to obtain the preprocessed historical log data;

[0039] Step S3: Obtain historical word vectors based on the preprocessed historical log data;

[0040] Step S4: Input the historical word vector into the Siamese LSTM network to extract the log pattern;

[0041] The matching of the log pattern is based on the log pattern obtained by the log pattern extraction method and the twin LSTM network, and the log pattern matching of the log data to be matched is performed.

[0042] in particular:

[0043] 1) Extract log mode

[0044] Such as image 3 As shown, for the historical log data of all components that appear in the system, first scan them one by one, perform simple data cleaning, and th...

Images