Massive log information processing method

Abstract

The invention discloses a massive log information processing method. A client-side filters acquired and newly increased log information, can decease transmission bandwidth occupied by invalid log information during log information transmission, sends the log information having high real-timeliness to a server in real time, delays sending of the log information having low real-timeliness to a server and can effectively shorten a log information acquisition period, a log server sends alarm log information to an alarm processing device in real time, can meet the requirement for timely output of the alarm information in log information and respectively stores non-alarm log information to different storage positions, and the categories of the log information correspond to operation categories for operating the log information so that the storage positions of the log information can be judged according to the operation categories in the log information operating process of a log operation platform, the processing time for log data operation can be rapidly and effectively shortened, and the processing efficiency can be improved.

Description

Technical field [0001] The present invention relates to data processing technology, in particular to a method for processing massive log information. Background technique [0002] For the business support network operation management system, as the complexity and diversification of the support network and business gradually increase, the granularity of business monitoring is getting finer and finer, and the resulting service log information volume is increasing, especially The data volume of the business service log is larger. The business service log of a communications operator may exceed 200G a day, and these business service logs are distributed on dozens of servers. At the same time, the business monitoring center has increasingly higher requirements for real-time alarm information. How to quickly complete log collection, refining, processing, storage, and efficient query has become the focus of the business support network operation management system. [0003] Currently, ...

AI Technical Summary

Problems solved by technology

[0017] According to the technical solution of the embodiment of the present invention, in the method for processing massive log information for log information collection provided by the embodiment of the present invention, the log client can filter the newly added log information collected, and can filter out the collected log information. Invalid log information in the log information, reduce the transmission bandwidth occupied by invalid log information when transmitting log information, and according to the corresponding relationship between predetermined log information and real-time performance level, carry real-time identification in the filtered log information, in In the process of sending the filtered log information to the server, the log information with high real-time performance is sent to the server in real time, and the log information with low real-time performance is sent to the server later, which can effectively shorten the cycle of log information collection; In the method for processing massive log information for storing log information provided by the embodiments of the invention, the log server sends the alarm log information to the alarm processing device in time, and can quickly send the alarm information to the alarm processing device, which can meet the requirements for log information. According to the requirement of timely output of the alarm information in the system, according to the category of the non-alarm log information and the time of receiving the log information, the non-alarm log information is stored in the relational database, the memory database of the server or the file system respectively, and the log information The category corresponds to the operation category for operating on the log information, so that the log operation platform can determine the storage location of the log information according to the operation category during the operation of the log information, and the relational database and the memory database can store data The reading speed is fast, which can quickly and effectively shorten the processing time for log data operations and improve processing efficiency, so as to solve the problem of collecting, processing and storing query processing methods for massive log information in the prior art, and the collection cycle is too long , Invalid log information occupies too much transmission bandwidth, the operation and processing time of massive log information is long, the efficiency is low, and the problem of timely output of alarm information in log information cannot be met

Images