Intelligent analysis system and method for big data logs

Abstract

The invention discloses an intelligent analysis system and method for big data logs, and relates to the field of big data log analysis. The system comprises a log acquisition module, a log input module used for distributing log stream message queues to hosts of a log analysis module through a load balancer, the log analysis module, a log analysis processing module, a log enriching module, a log storage module, a log retrieval display module, a log output module and an alarm module. According to the intelligent analysis system and method for the big data logs provided by the invention, the analysis system has an analysis function in combination with an operation and maintenance scene; correlation analysis of service logs and performance logs is achieved; the workload of secondary development is small; and an enterprise can be helped to solve the practical operation and maintenance problem through the log analysis system.

Description

technical field [0001] The invention relates to the field of big data log analysis, in particular to a big data log intelligent analysis system and method. Background technique [0002] There are many log solutions, such as Splunk, Facebook's Scribe, Apache's Chukwa, LinkedIn's Kafka, Cloudera's Flume, and the combination of Kibana+Elasticsearch+Logstash. The main architecture modules of these log solutions are similar, including log collection and transmission modules, log collection Storage module and log parsing search module. These large and comprehensive log solutions in the current business environment solve the needs of enterprises without a big data log analysis system. The implementation of the log collection and transmission module of the existing log solutions is to deploy Agent probes on the log target host. The collected log data is transmitted to the log collection and storage module through snmp and other network protocols. The log collection and storage modu...

AI Technical Summary

Problems solved by technology

These large and comprehensive log solutions in the current business environment solve the needs of enterprises without a big data log analysis system. The implementation of the log collection and transmission module of the existing log solutions is to deploy Agent probes on the log target host. The collected log data is transmitted to the log collection and storage module through snmp and other network protocols. The log collection and storage module distributes and stores the original logs in each log storage server through load balancing, and establishes a log keyword index at the same time. The log analysis and search module will log Do further analysis, filtering, sorting and other operations to facilitate log users to retrieve and view. The existing log solution is mainly a complete set of large and comprehensive log collection, storage and retrieval framework. If an enterprise wants to use the existing log solution To provide help for daily operation and maintenance work, enterprises need to do secondary development according to their actual situation, because this type of log analysis system does not have the analysis function of combining logs with operation and maintenance scenarios, and lacks the correlation analysis of business logs and performance logs, so secondary development The workload is very heavy, so even if this kind of log system is built, the enterprise still has a lot of work to solve the actual operation and maintenance problems and fault location through the log system, the in-depth analysis of the log and the actual operation and maintenance monitoring system, etc. The lack of functions leads to the failure to reflect and effectively utilize the value of logs

Images