System and method for high-speed memory and distributed type processing of massive logs

Abstract

The invention discloses a system and a method for high-speed memory and distributed type processing of massive logs. The system comprises a plurality of log generating devices and a log management server, wherein the log generating devices are dispersed in all network elements and are used for carrying out format normalization on various original logs, caching the original logs into high-speed memory components and uploading the original logs to the log management server periodically; each log generation device is respectively provided with three parts, i.e., an original log generation module, a log pre-processing module and a high-speed memory component. The log management server is responsible for unified derivation and filing storage of the logs beyond the caching period in the high-speed memory components distributed in all the log generating devices so as to be used for inquiring and analyzing the logs by other systems. The system and the method disclosed by the invention have the beneficial effects that a plurality of types of logs can be processed in real time with high-efficiency and low cost, and both the expandability and the actionability are better. Compared with the mode of adopting a relational database to store the logs, the system and the method disclosed by the invention have the advantages that the processing cost is reduced, the bottleneck problem of the database with large data volume is also solved, and the efficiency of querying and analyzing the logs is improved.

Description

technical field [0001] The invention relates to a communication log processing technology, more precisely, to a system and method for distributing and processing massive logs through high-speed memory, and belongs to the field of communication technology. Background technique [0002] In the service network of a telecom operator, the communication service log for network managers to understand the service usage status is an indispensable part. Especially with the continuous evolution and development of factors such as network diversification, IP, and business openness, the security threats faced by telecommunications equipment are also increasing. It is possible to record and analyze logs of user business usage in real time, and to discover and locate network The role in the problem is also becoming more and more important. Moreover, with the rapid expansion of network services and the increasing number of users, the types and quantities of telecommunications network equipm...

AI Technical Summary

Problems solved by technology

However, for log archiving and backup, especially when providing external log query and analysis, this processing method must poll the message queue, which is inefficient

Images