Method and device for abnormity detection and processing of log files

Abstract

The invention discloses a method for abnormity detection and processing of log files. The method comprises the following steps: 1) abnormity monitoring and processing of a BI log server log and a data source MD5: (1) comparing the BI log server log with the data source MD5 and detecting whether the BI log server log and the data source MD5 are abnormal, and (2) processing the abnormal condition that a BI log server log file is inconsistent with a data source MD5 file; 2) abnormity monitoring and processing of the BI log server log file and the data source MD5 file: (1) detecting whether the BI log server log file is abnormal, (2) identifying content in the file, and when records not under the date of a folder exist in the file under the folder or records under the date and records not under the date coexist in the file, considering that the log file has packaging error, and (3) abnormity processing of the BI log server log file: carrying out abnormity detection on the BI log server log file and carrying out processing after abnormity is found; and 3) abnormity detection and processing of an HDFS file.

Description

technical field [0001] The invention relates to the field of website log analysis, in particular to a method and device for abnormal detection and processing of log files. Background technique [0002] The rapid development of Internet technology has allowed websites to accumulate even larger access logs. A complete and comprehensive analysis of access logs can mine user behavior characteristics, improve the website's own operating efficiency and obtain greater profits. However, since the access log files are huge and scattered, and the log files need to be transferred and stored several times during the entire analysis process, the integrity and accuracy of the log files cannot be guaranteed during this process. Taking Made in China (hereinafter referred to as MIC) as an example, the entire log analysis process and possible problems can be summarized as follows: [0003] 1. Every morning, the operation and maintenance department packs the previous day's logs and stores th...

AI Technical Summary

Problems solved by technology

In this process, the final result may be inaccurate due to a program bug

[0006] If the integrity and accuracy of the log files cannot be guaranteed, it will cause inaccurate analysis of user behavior by the website, which will affect the analysis effect and even the operation effect of the entire website.

Images