Massive log data intelligent operation and maintenance system

Abstract

The invention provides a massive log data intelligent operation and maintenance system. The system comprises a log collecting module, a log processing module, a log storage module and a business application module, wherein the log collecting module is used for collecting application logs, operation system logs and network equipment logs on a target machine, structuring log data and then outputtingthe log data to the log processing module; the log processing module is used for real-time processing and offline processing, real-time processing comprises real-time statistics, real-time alarming and real-time analysis, and offline processing comprises data batch storage, offline analysis and machine learning; then the processed information is sent to the log storage module; the log storage module is used for storing original log data, index data, statistical analysis data and resource configuration data, and comprises a MySQL relation database, a Redis memory database, an Elastic search engine and an HBase column storage database; the business application module is used for providing functions of inquiry statistics, monitoring and alarming, intelligent reports, instrument panels and knowledge bases through a web interface, and the data is read out from the log storage module.

Description

technical field [0001] The present invention relates to the field of Internet application technology, to massive log processing technology, and in particular to an intelligent operation and maintenance system for massive log data. Background technique [0002] As the volume and types of log files grow, it becomes more and more difficult to analyze log data, track down potential problems, and find errors, especially after the emergence of multi-log correlation analysis. Even at the best of times, it takes experienced operators to follow the chain of events, filter the noise, and ultimately diagnose the root cause of complex problems. [0003] Traditional operation and maintenance products need to log in to each server and use script commands or programs to view. The operation is cumbersome and error-prone. The data is isolated and scattered, and cannot be correlated or extracted. Statistics cannot meet the analysis requirements, and there is no real-time monitoring and alarm...

AI Technical Summary

Problems solved by technology

[0004] In order to solve the current situation of large amount of massive log data, isolated data, and difficult analysis under the framework of the existing technology, the present invention proposes a massive log intelligent operation and maintenance system

Images