Method and device for abnormity detection and processing of log files

A log file and anomaly detection technology, applied in digital data processing, error detection/correction, hardware monitoring, etc., can solve inaccurate analysis, affect website analysis effect and even operation effect, and log file integrity and accuracy cannot be obtained Guarantee and other issues to achieve the effect of analysis improvement

Active Publication Date: 2016-06-22
FOCUS TECH
View PDF2 Cites 21 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In this process, the final result may be inaccurate due to a program bug
[0006] If the integrity and accuracy of the log files cannot be guaranteed, it will cause inaccurate analysis of user behavior by the website, which will affect the analysis effect and even the operation effect of the entire website.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for abnormity detection and processing of log files
  • Method and device for abnormity detection and processing of log files
  • Method and device for abnormity detection and processing of log files

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0034] figure 1 A flowchart showing a log file abnormality detection and processing method according to an embodiment of the present invention.

[0035] A method and device for abnormal detection and processing of log files, comprising:

[0036] 1), BI log server log and data source MD5 exception monitoring and processing

[0037] (1) Compare the BI log server log with the MD5 of the data source to detect whether it is abnormal

[0038] When the operation and maintenance package mounts the data source log file and the BI log server downloads the log, the MD5 value is generated for each file with the same MD5 encryption method and stored in two tables respectively. The MD5 value of the data source file and the original BI log server By comparing the MD5 value of the log file, you can judge whether the BI log server log is consistent with the data source file, and give corresponding warnings for abnormal situations.

[0039] (2) Handle the abnormal situation that the log file...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method for abnormity detection and processing of log files. The method comprises the following steps: 1) abnormity monitoring and processing of a BI log server log and a data source MD5: (1) comparing the BI log server log with the data source MD5 and detecting whether the BI log server log and the data source MD5 are abnormal, and (2) processing the abnormal condition that a BI log server log file is inconsistent with a data source MD5 file; 2) abnormity monitoring and processing of the BI log server log file and the data source MD5 file: (1) detecting whether the BI log server log file is abnormal, (2) identifying content in the file, and when records not under the date of a folder exist in the file under the folder or records under the date and records not under the date coexist in the file, considering that the log file has packaging error, and (3) abnormity processing of the BI log server log file: carrying out abnormity detection on the BI log server log file and carrying out processing after abnormity is found; and 3) abnormity detection and processing of an HDFS file.

Description

technical field [0001] The invention relates to the field of website log analysis, in particular to a method and device for abnormal detection and processing of log files. Background technique [0002] The rapid development of Internet technology has allowed websites to accumulate even larger access logs. A complete and comprehensive analysis of access logs can mine user behavior characteristics, improve the website's own operating efficiency and obtain greater profits. However, since the access log files are huge and scattered, and the log files need to be transferred and stored several times during the entire analysis process, the integrity and accuracy of the log files cannot be guaranteed during this process. Taking Made in China (hereinafter referred to as MIC) as an example, the entire log analysis process and possible problems can be summarized as follows: [0003] 1. Every morning, the operation and maintenance department packs the previous day's logs and stores th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/30
CPCG06F11/3006G06F11/3072
Inventor 邹凤莲陈静
Owner FOCUS TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap