A streaming on-line log analysis method

A parsing method and log technology, applied in structured data retrieval, file system types, database management systems, etc., can solve the problem that it is difficult for users to determine how to set up, and achieve the effect of fast speed, high efficiency, and robust log

Active Publication Date: 2019-01-11
XI AN JIAOTONG UNIV
View PDF5 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, this method still needs to be greatly improved in accuracy, and the depth of the tree is often difficult for users to determine how to set it as a parameter.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A streaming on-line log analysis method
  • A streaming on-line log analysis method
  • A streaming on-line log analysis method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0036] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0037] In the present invention, unstructured log text is parsed into structured log type.

[0038] see Figure 6 , the present invention comprises the following steps:

[0039] 1) Preprocess the log text (remove the columns that must be parameters such as timestamp);

[0040] The constant part and the variable part are defined as follows:

[0041] The part of the log that can be used as a log type is called a constant part, and the part that cannot be used as a log type is called a variable part. Such as figure 1 Shown:

[0042] figure 1 Medium logs 1-6 are 6 logs from different systems, and log types 1-4 are log types corresponding to logs 1-6. Among them, the non-italic part is the constant part, that is, the log type; the italic part is the variable part, that is, the parameter.

[0043] 2) Match partition: according to log m i length to search whether t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a streaming on-line log analysis method, which partitions the log according to its length by using the idea of partition, that is, the log of the same length is distributed tothe same partition. After the log partition, the fast matching phase determines whether the log belongs to the current log type by calculating whether the intersection of the log and the log type meets a threshold. After the log type to which the log belongs is quickly matched, the log type extraction phase extracts the log type and log parameters by finding the longest common sub-sequence of thelog and log type. The method can effectively parse the system log and parse the log from unstructured text to structured log type, and the result can be used for abnormal detection of the log, and themethod is simple and effective.

Description

technical field [0001] The invention belongs to the field of system log safety monitoring and relates to a streaming online log parsing method. Background technique [0002] The analysis of system logs from unstructured text to structured log types has important applications in the fields of system log security monitoring and network anomaly detection. According to the results of log analysis, the running sequence of programs in the system can be clearly understood, and further can be used It is used in the construction of program workflow and abnormal detection in the system; the classic log parsing method can parse the log more effectively, and the existing methods can be divided into two categories: offline method and online method: offline means that the log needs to be collected for a period of time Generated logs, analyze these logs at one time, if there are newly generated logs that need to be parsed, then retraining is required; the online method does not need to loa...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25G06F16/18
Inventor 王晨旭赵志远饶巍陶敬马小博秦涛
Owner XI AN JIAOTONG UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap