High-performance Syslog processing and storage method

Abstract

The invention is high performance Syslog log processing and storing method, and the steps are: (1) receiving log: log sever software receives the syslog data message by separated thread running method via bound UDP port (514), and the data message is generated into data class of syslog, and the data class of syslog is written in log buffer; (2) log buffer: the log buffer takes charge of storing the received syslog log data temporarily; (3) log normalization processing: detects the head of log buffer by separated thread, if the log buffer has new data, and the data will be extracted from buffer queue, and log will execute normalization processing according to field description information of log normalization configuring file, and call the log storing module to store the log as uniform format; (4) log storage: using the storing thread pool to write said normalization log on log storage file, every thread takes charge of writing log data in one file, and several threads can write the logs in different directories and discs, and improve the storing efficiency. The technical schedule adopts asynchronies log processing and intercurrent log storage technique, and two taches can inherit each other and coordinates the problems of log amount and log inercurrent amount, and use time to change time, and improve the log processing ability greatly.

Description

technical field [0001] The invention relates to a high-performance Syslog log processing and storage method, which relates to computer system integration and application technology, and in particular to the syslog log processing and storage technology in network information security. Background technique [0002] With the increasing scale of the network environment, the number of various devices in the network has increased sharply, and various security and attacks from the outside and inside have also increased sharply, threatening the security of network information. Therefore, security audits have become extremely important. The log of the device records detailed information such as the operating status of the device, the operations performed by various users, and so on. In the current network environment, the logs of various devices have become massive data. As the main log type, syslog is widely supported by various operating systems, network devices and security device...

AI Technical Summary

Problems solved by technology

[0006] Log files are stored quickly and are easy to manage, but the query performance is poor. Since there is no index, the query efficiency is relatively low, and it is not convenient to query and analyze the logs

Images