System and method for monitoring and analyzing great deal of logs in real time

Abstract

The invention discloses a system and method for monitoring and analyzing a great deal of logs in real time, and belongs to the technical field of webmaster monitoring. The system for monitoring and analyzing a great deal of logs in real time comprises a log collecting module, a log summarizing and distributing module and a log processing module; the log collecting module adopts an Flume-ng technology and is divided into an Agent layer, a Collector layer and a Store layer; the log summarizing and distributing module adopts a kafka technology and is used for receiving log information of Flume clusters and distributing the received log information to the log processing module; the log processing module adopts a storm technology and is composed of a data source Spout and a log processor Bolt. The system for monitoring and analyzing a great deal of logs in real time can process risks and faults in a network in real time to ensure safety and stability of the network and reduce loss of a user and has excellent application and popularization value.

Description

technical field [0001] The invention relates to the technical field of network management monitoring, and specifically provides a system and method for real-time monitoring and analysis of a large number of logs. Background technique [0002] With the rapid development of Internet technology, the scale of the network is getting bigger and bigger, and the network is becoming more and more complex. In order to ensure the normal operation of the network, real-time monitoring of network equipment and application systems in the network is required. [0003] Due to the complexity of the network, the forms of log messages are also varied. Traditional log monitoring methods need to develop and customize a large number of interfaces, which will waste a lot of human and financial resources. In addition, the traditional log monitoring method has many shortcomings in real-time processing of massive logs. For example, when a new network device or application system is added to the netwo...

AI Technical Summary

Problems solved by technology

Traditional log monitoring methods need to develop and customize a large number of interfaces, which will waste a lot of human and financial resources

In addition, the traditional log monitoring method has many shortcomings in real-time processing of massive logs. For example, when a new network device or application system is added to the network, it is necessary to customize and develop corresponding log collection for the newly added device or system. Script; the traditional method is to store the collected log messages on the disk first, and then distribute them to the log processing engine for processing, which has poor timeliness and cannot meet the requirements of real-time processing; for log messages in text mode, when processing log messages , it is difficult to detect which logs are newly generated. The traditional method needs to scan the entire log file to determine the latest log messages. In the face of massive log messages, it is impossible to complete log processing on one machine, and It is necessary to use a distributed processing method, and the traditional method is mostly concentrated on one server, which cannot meet the requirements of massive log processing; since the processing of logs involves multiple processing links, not only the distributed processing method is required, but also the The stream processing method, the traditional log processing method is difficult to meet this requirement

[0004] When dealing with massive logs in the traditional way, the analysis and processing lags behind, and it is impossible to achieve real-time analysis and processing, and it is impossible to understand and deal with risks and faults in the network in time, which brings great inconvenience to the company and customers

Images