Log parsing method and device based on perfect hashing

Abstract

The embodiment of the invention provides a log parsing method and device based on perfect hashing. The method includes the steps of obtaining a to-be-parsed log, conducting information extraction on the to-be-parsed log according to a preset rule, and obtaining to-be-parsed keywords and corresponding values of the to-be-parsed log; according to a perfect hash function created in advance, calculating a first hash value corresponding to each to-be-parsed keyword; according to each first hash value, obtaining corresponding to-be-parsed keyword types, saving the corresponding value of each to-be-parsed keyword into a corresponding structure of the corresponding to-be-parsed keyword type for parsing to obtain structural data. The device is used for executing the method. According to the embodiment, by adopting a hash locating method for identifying the to-be-parsed keywords, there is no need to match characters of the to-be-parsed keywords one by one, the keywords can be obtained fast through hash locating, and therefore the efficiency of parsing the log is improved.

Description

technical field [0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a perfect hash-based log parsing method and device. Background technique [0002] Any program in the computer system may output logs: the operating system kernel, various application servers, and so on. The log contains a large amount of information that is of interest to security managers, operation and maintenance personnel, and business analysts, such as the visitor's IP, access time, source address, client information used by the visitor, and analysis of user behavior characteristics, etc. . [0003] Most logs are strings, and a fixed position represents a fixed field. Take firewall logs as an example. Logs in this string format cannot meet the needs of firewall log content changing within a certain range. Therefore, key-value pairs are used on the firewall. After the monitoring device receives the log sent by the firewall, the traditional me...

AI Technical Summary

Problems solved by technology

[0003] Most logs are strings, and a fixed position represents a fixed field. Take firewall logs as an example. Logs in this string format cannot meet the needs of firewall log content changing within a certain range. Therefore, key-value pairs are used on the firewall. After the monitoring device receives the log sent by the firewall, the traditional method is to use string traversal to find out each pair of key and value. After obtaining the key, obtain the type of the keyword through keyword matching, where the keyword matches The time is proportional to the length of keywords. When there are many keywords, the processing efficiency is low, which seriously affects the parsing efficiency of firewall logs.

Images