Word segmentation algorithm-based log parsing method and word segmentation algorithm-based log parsing system

Abstract

The invention relates to the technical field of log audit and safety management, and aims at providing a word segmentation algorithm-based log parsing method and a word segmentation algorithm-based log parsing system. The word segmentation algorithm-based log parsing method comprises the following steps: performing segmentation on a log, performing word sense analysis on segmentation results, performing word sense filtration on obtained segmentation results with word sense tagging, performing feature extraction on the obtained filtered segmentation results with the word sense tagging, performing feature matching on obtained word sense order feature codes, and performing semantic parsing on obtained semantic parsing rules; the word segmentation algorithm-based log parsing system comprises a segmentation module, a word sense analysis module, a word sense filtration module, a word order feature extraction module, a feature matching module and a semantic parsing module. According to the word segmentation algorithm-based log parsing method and the word segmentation algorithm-based log parsing system disclosed by the invention, the difficulty and complexity of log parsing are greatly reduced, and therefore the efficiency of performing parsing rule development on the log is increased; the word segmentation algorithm-based log parsing method and the word segmentation algorithm-based log parsing system can be better adapted to certain changes of a log format.

Description

technical field [0001] The invention relates to the technical field of log auditing and safety management, in particular to a log parsing method and system based on a word segmentation algorithm. Background technique [0002] Any program in the computer system may output logs: the operating system kernel, various application servers, and so on. The log contains a lot of information that people—mainly security managers, operation and maintenance personnel, and business analysts—will be interested in, such as the visitor's IP, access time, source address, and client information used by the visitor, Analyze user behavior characteristics, etc. [0003] Since these logs are so useful, how to perform log analysis is not a simple problem. Logs contain thousands of possible formats and data, and "analysis" is even more difficult to define. It may be simple calculation of statistical values ​​or complex data mining algorithms. Of course, there are countless ready-made tools that c...

AI Technical Summary

Problems solved by technology

The approach based on regular expressions may give us headaches after a period of time, especially how to maintain a large number of regular expressions, mismatches between regular expressions, and low performance of regular expression chain matching.

Images