Log analysis method based on syslog protocol

Abstract

The log analysis method based on syslog protocol comprises the following steps: modifying the data reading logic of the log analysis tool, listening to the original log data of the network protocol from the TCP / UDP interface of the syslog client; based on the syslog protocol specification, analyzing the log data information by itself, and performing the analysis and processing of the log data on the original log data analysis logic in the interface program; based on the analyzed log content, filtering and deduplicating according to the category, reducing the log data volume, and performing the cache of a certain amount of log data. The method for directly reading the log data from the TCP / UDP interface of the syslog client can ensure that the log data is analyzed without omission, the log analysis effect is not reduced, the log data does not need to be written to the disk, the disk IO and disk space resources are not consumed in a large amount, and the stability of the system is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of log data analysis, and particularly relates to a log analysis method based on a syslog protocol. BACKGROUND

[0002] Syslog is a widely used log protocol in Unix-like systems, which is used for recording system events, including the running status of the core and system programs and the events occurred. Syslog is supported by almost all network devices, such as routers, switches and printers. The RFC standards of syslog include RFC 3164, RFC 5424 and RFC 3195. The first two standards use UDP protocol, and the main difference lies in the format of data. The third standard uses TCP protocol to provide reliable data transmission, but since the data transmission based on UDP is stable enough in actual use scenarios, the protocol RFC3195 is less used in practice.

[0003] In the current mainstream Linux operating system, systemd is the default syslog client end, which exposes system logs through Unixsocket or socket port (514); rsyslog is the default syslog server end, which can collect and record system logs for users to analyze log data and troubleshoot problems.

[0004] In the actual use scenario of Linux servers, a large amount of log data of various types is transmitted to the unified log center based on systemd and exposed to the outside through the syslog protocol. A typical example is the log data of user facility, i.e. the log data generated by user processes. In the default configuration of the service, if the process outputs the log to stdout by default, the log will finally be output by systemd through syslog, and finally recorded by rsyslog to the system log file. In the case of abnormal business process, a large amount of log data may be output in a short time, resulting in a large amount of disk IO occupation and affecting business and system performance. However, if such logs are not recorded, the part of business log data will be lost, making it difficult to analyze business failures. In addition, after connecting with log data analysis tools, the log data file will also be read and analyzed. The storage and reading of a large amount of log data will result in a large amount of IO resource consumption, and also cause a large amount of disk resource waste, which may even lead to disk space exhaustion and system and business crash in extreme cases.

[0005] As Figure 1As shown, the conventional log analysis tool reads data from the disk log file, thereby inevitably causing storage and re-reading of the log data, which wastes disk IO and disk space resources. SUMMARY

[0006] To solve the problems in the prior art, the present application provides a log analysis method based on a syslog protocol, comprising the following steps:

[0007] Step S1: modifying the data reading logic of the log analysis tool to listen to raw log data of the network protocol from the TCP / UDP interface of the syslog client;

[0008] Step S2: based on the syslog protocol specification, analyzing and processing the log data by analyzing the log data information and the original log data analysis logic in the interface program;

[0009] Step S3: based on the parsed log content, filtering and deduplicating according to the category to reduce the amount of log data and performing a certain amount of log data caching.

[0010] In step S1, the log analysis tool directly interfaces with the syslog client from the / dev / log Unix socket file to listen to TCP / UDP data and read the log raw data.

[0011] In step S2, the log data based on the UDP protocol is parsed based on the syslog RFC 3164 and RFC 5424 specifications, or the log data based on the TCP protocol is parsed based on the RFC 3195 specification, and the final log data information includes the facility, time, level, host, process, and content details of the log data.

[0012] In step S3, the following steps are included:

[0013] Step S31: checking whether the log is the target to be monitored according to the facility, process, and log level information of the log, and if not, discarding it, otherwise, continuing to process;

[0014] Step S32: comparing the log content with the previously cached log data to check whether the log data is a repeated log, and if so, discarding it, otherwise, continuing to process;

[0015] Step S33: checking whether the amount of cached log data reaches the maximum data limit, and if so, performing rolling deletion to discard the earliest recorded log data, otherwise, continuing to process.

[0016] The method further comprises a step S4 of closing the additional data monitoring and disk dumping.

[0017] The method of directly reading log data from the TCP / UDP interface of the syslog client can ensure that the log data is analyzed without omission and the log analysis effect is not reduced, and the log data does not need to be dumped, so that the disk IO and disk space resources are not consumed in large quantities, thereby avoiding the large consumption of disk IO and disk space resources in the case of incorrect rsyslog configuration and abnormal business log output, and improving the stability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The method of reading data from a disk log file by an existing log analysis tool.

[0019] Figure 2 The log analysis method based on the syslog protocol of the present application.

[0020] Figure 3 The log data processing flowchart in the log analysis method based on the syslog protocol of the present application. DETAILED DESCRIPTION

[0021] In order to have a further understanding of the technical solutions and beneficial effects of the present application, the technical solutions of the present application and the beneficial effects thereof will be described in detail below with reference to the accompanying drawings.

[0022] In order to avoid the problem of resource consumption of disk IO and disk space caused by log data dumping and improve the overall stability of the system, the present application provides a new log data analysis method, which bypasses the traditional method of using an rsyslog server to dump log data and then analyzing log text files, directly reads log data from the TCP / UDP interface of the syslog client, and does not need to store log data in files, thereby avoiding the disk IO occupation and disk resource consumption caused by data dumping and shortening the log analysis process.

[0023] The key terms involved in the present application are explained as follows:

[0024] rsyslog: a server that implements the syslog protocol, which can be used for collection, conversion, forwarding, storage, etc. of server system logs.

[0025] Dumping: refers to storing data into a server disk / hard disk or other storage medium.

[0026] The technical means adopted by the present application is: the present application is through self-research log analysis tool, its working logic lies in through the method of optimizing log data acquisition process, skips rsyslog's log data collection and data landing, directly obtains system log data from syslogclient's TCP / UDP interface, analyzes log data, on this basis, carries out filtering and analysis to log output, so as to not need to land on the log data, avoids the consumption of disk IO and disk space resources.

[0027] The method for realizing it is as follows:

[0028] 1, modify the data reading logic of log analysis tool, and listen to the original log data of network protocol from the TCP / UDP interface (such as / dev / log) of syslog client.

[0029] 2, based on syslog protocol specification, self-analyzing log data information, and analyzing and processing the original log data analysis logic in the interface program.

[0030] <34>Oct 10 20:13:14 mymachine program:To be,or not to be,that is the question.

[0031] In turn, it indicates:

[0032] Priority number: <34>, calculated from facility and severity

[0033] Timestamp: timestamp information Oct 10 20:13:14

[0034] Host name: host name, mymachine

[0035] Tag: usually process name program name, when it has PID, display program [PID]:

[0036] Message: log content details, hello world.

[0037] 3, Based on the parsed log content, according to the category (including but not limited to facility, process, level, content, etc.) filtering, deduplication, etc., reduce the amount of log data, and cache a certain amount of log data. Since the data exposed by the TCP / UDP interface of syslogclient does not have persistence, the data reading module (or the whole) of the log analysis tool needs to be continuously monitored as a system service, and the system core log data needs to be retained for local analysis when necessary, so that the system failure has a certain traceability,

[0038] 4, Close part or all of the log disk configuration in the rsyslog program configuration, or close the rsyslog service, and completely close the log data disk.

[0039] Figure 2 For the overall implementation flowchart of the present application, as shown in Figure 2 A specific embodiment of the present application is as follows:

[0040] Step 1: Log analysis tool initialization.

[0041] As shown in the following Figure 2 The log analysis tool no longer reads log text data from the disk log file, but directly interfaces with syslog client from / dev / log Unix socket file, listens to TCP / UDP data, and reads log raw data.

[0042] Step 2: Log data parsing.

[0043] Based on the RFC 3164, RFC 5424 specification of syslog, parse the log data based on UDP protocol, or parse the log data based on TCP protocol based on RFC3195 specification, and finally get the facility, time, level, host, process, content details and other information of the log data.

[0044] Step 3: Cache and filter log data.

[0045] According to the configuration of the software, the parsed data is filtered and processed. For example Figure 3The shown: 1) according to the facility, process, log level and other information of the log, check whether this log is the target of monitoring, if not, discard, otherwise continue to process; 2) compare the log content with the previously cached log data, check whether this log data is a repeated log, if so, discard, otherwise continue to process; 3) check whether the amount of cached log data reaches the maximum data limit (number or byte number), if so, perform rolling deletion, discard the earliest recorded log data, otherwise continue to process; 4) filter the processed data, which is of high value, and manually view and analyze through the terminal or transmit to other log analysis platforms for further automatic analysis.

[0046] For the data of the first three processing links, statistical log data total, repeated log content, discarded log data amount and cache usage, etc. Statistical information is obtained to compensate for the information deviation caused by data discarding, and log analysis tool parameter optimization is performed.

[0047] Step 4 (optional): Close additional data monitoring and disk landing.

[0048] Close the data landing of rsyslog, reduce the consumption of disk IO and disk space.

[0049] The beneficial technical effect of the present application is that the method of directly reading log data from the TCP / UDP interface of syslog client can ensure that the log data is analyzed without missing and the log analysis effect is not reduced, and the log data does not need to be landed, which ensures that the disk IO and disk space resources will not be consumed in large quantities, thereby avoiding the large consumption of disk IO and disk space resources in the case of incorrect rsyslog configuration and abnormal business log output, and improving the stability of the system.

[0050] Although the present application has been described by the above preferred embodiments, it is not intended to limit the protection scope of the present application, and any person skilled in the art can make various changes and modifications to the above embodiments without departing from the spirit and scope of the present application, which are still within the scope of protection of the present application, therefore the protection scope of the present application is defined by the claims.

Claims

1. A log analysis method based on the syslog protocol, characterized in that, Includes the following steps: Step S1: Modify the data reading logic of the log analysis tool. The log analysis tool directly connects to the syslog client from the / dev / log Unix socket file, listens for TCP / UDP data, and reads the raw log data. Step S2: Based on the syslog protocol specification, parse the log data information independently, connect it with the original log data analysis logic in the program, and perform log data analysis and processing; Step S3: Based on the parsed log content, filter and deduplicate according to category to reduce the amount of log data, and cache a certain amount of log data; Step S4: Disable additional data monitoring and disk write.

2. The log analysis method based on the syslog protocol as described in claim 1, characterized in that, In step S2, log data based on the UDP protocol is parsed according to the RFC 3164 and RFC 5424 specifications of syslog, or log data based on the TCP protocol is parsed according to the RFC 3195 specification. The final log data information includes the facility, time, level, host, process and content details of the log data.

3. The log analysis method based on the syslog protocol as described in claim 1, characterized in that, Step S3 includes: Step S31: Based on the log's facility, process, and log level information, check whether this log entry is the target of monitoring. If not, discard it; otherwise, continue processing. Step S32: Compare the log content with the previously cached log data to check if the current log data is a duplicate. If it is, discard it; otherwise, continue processing. Step S33: Check if the amount of cached log data has reached the maximum data limit. If so, perform rolling deletion and discard the earliest recorded log data; otherwise, continue processing.

Images