A syslog log automatic parsing method

Abstract

This invention discloses an automatic Syslog log parsing method, comprising: collecting Syslog log messages; removing header information and retaining the message body content, parsing it into a string set, and determining a delimiter; using the delimiter to segment the Syslog log message body content into a string array; confirming whether a key exists in the string array; classifying the strings, if a key exists, classifying the logs according to the key and data length; if no key exists, checking the data type and classifying the logs according to the array length and data type; automatically generating parsing templates for different log categories, using the parsing templates to extract the value corresponding to each field, and performing subsequent field mapping or transformation normalization processing. Using this invention, regular expression parsing of Syslog logs can be generated quickly, saving significant manpower.

Description

Technical Field

[0001] This invention relates to the field of computer information technology, and in particular to an automatic Syslog log parsing method. Background Technology

[0002] Syslog logs are the most common logging format in computer and network systems, generally conforming to RFC 3164 and RFC 5424 standards. Syslog logs are used to record critical processes during the operation of business systems or network devices, for auditing or problem analysis. Originally used in UNIX systems, Syslog is now supported by virtually all network devices. It is a standard protocol, and each vendor and each device has its own Syslog logging standard, recording key information during business processing. The Syslog protocol supports TCP or UDP, but generally uses UDP as the transport protocol, with the default receive port being 514.

[0003] Syslog logs originate from various hardware and software systems from different vendors, and their application scenarios differ, often lacking a unified Syslog format. For log auditing and analysis systems, parsing this multi-source, heterogeneous Syslog data is a crucial task. A common solution is to develop log parsing templates (usually regular expressions) for each different device model and software / system version, providing parsing capabilities for that specific log format. These regular expressions are defined as static templates, loaded when the log system's parsing module starts. During runtime, the log system matches each new log entry against the predefined log format parsing template, extracting all fields upon successful match. Currently, these templates, i.e., the regular expressions, are manually written, requiring manual analysis of each log entry, resulting in a significant workload for log matching and parsing.

[0004] Currently, there has been considerable research and exploration into automated parsing of unknown logs. For example, patent application CN202010379526.0 collects log messages and divides them into different log groups based on Levenshtein edit distance similarity. The structured information of the log message content is extracted based on the grouping results and merged with the extracted structured information from the log header to form the overall structured information of the log message. However, this technical solution lacks automation in the preprocessing stage, resulting in a significant workload for manual analysis. Furthermore, the solution lacks automatic delimiter recognition technology, which presents challenges in practical applications of Syslog parsing and fails to achieve satisfactory automation.

[0005] In view of the incompleteness of the existing solutions mentioned above, this invention constructs an automatic Syslog log parsing method that can fully integrate the prior knowledge of Syslog logs, extract features for common fields of Syslog logs, and has a simple algorithm that is easy to implement and engineer. Summary of the Invention

[0006] This invention provides a method for automatic parsing of Syslog logs, including:

[0007] Collect a large number of Syslog log messages from a single source;

[0008] Remove the header information from the Syslog log, retain the message body content, and parse it into a string collection;

[0009] Analyze the string set to determine the delimiter;

[0010] The Syslog log message body is split using a defined delimiter, and all logs are split into string arrays according to the delimiter;

[0011] Check if the string key exists in the string array;

[0012] The strings are categorized based on whether a key exists. If a key exists, the logs are categorized based on the key and the data length. If no key exists, the data type is checked, and the logs are categorized based on the array length and the data type.

[0013] Automatically generate parsing templates for different categories of logs, use the parsing templates to extract the value corresponding to each field, and perform subsequent field mapping or transformation normalization processing.

[0014] As described above, one method for automatically parsing Syslog logs involves a program on a PC receiving Syslog logs and saving them to a database or file.

[0015] The Syslog log automatic parsing method described above includes a set of delimiters including @='”?.: / \.

[0016] The Syslog log automatic parsing method described above uses a frequent itemset algorithm to find frequent sequences of delimiters for the set of delimiters listed in this application. These sequences are sets of single or multiple characters.

[0017] The Syslog log automatic parsing method described above uses an algorithm to check if a key exists in a string array: check if each string in the string array contains an equal sign or a colon.

[0018] The present invention also provides a computer storage medium, comprising: at least one memory and at least one processor;

[0019] The memory is used to store one or more program instructions;

[0020] A processor for running one or more program instructions to execute a Syslog log automatic parsing method as described above.

[0021] The beneficial effects achieved by this invention are as follows:

[0022] 1) By adopting the technical solution of this invention, regular expression parsing of Syslog logs can be generated quickly, saving a lot of manpower.

[0023] 2) By fully utilizing the Syslog protocol and industry best practices, the algorithm is more targeted, easier to implement, and simpler and more efficient. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0025] Figure 1 This is a flowchart of an automatic Syslog log parsing method provided in Embodiment 1 of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Example 1

[0028] Before introducing the automatic Syslog log parsing method provided in this application, the Syslog log attributes are described as follows:

[0029] (a) Syslog is the most common log format in computer and network systems, generally following the RFC3164 and RFC 5424 standards. Syslog logs are readable character-based strings with a unified header structure, while the specific content information is defined by each vendor. For example, the RFC 5424 standard specifies that the maximum message length is 2048 bytes.

[0030] (ii) The Syslog protocol explicitly defines the header structure, for example, the header structure is as follows: <30> Oct 922:33:2010.20.21.66auditd

[1787] , there are no specific requirements for the message body, which can be defined by each manufacturer.

[0031] Below are some typical Syslog log examples:

[0032] ① Sample log of vendor A switch:

[0033] <188> Mar 30 2018 10:12:17L13_HLGCLD_BSW01_7703%%01SECE / 4 / PORT_ATTACK_OCCUR(l)

[1001] :Autoport-defend started.(SourceAttackInterface=GigabitEthernet2 / 0 / 13,AttackProtocol=ARP-REQUEST)

[0034] ② Example of firewall logs from vendor A:

[0035] <188> Mar 20 2021 13:32:27HuaweiFW%%IPSEC / 5 / IPSEC_TUNNEL_ESTABLISHED:Vsys[vsys-name]:An IPSec tunnel is established.(PolicyName=policy-aaa,IfIndex=5,SeqNum=218,RuleNum=rule-aaa,SrcIP=192.168.2.6,DstIP=192.168.2.8,Slot=2,CpuID=2,State=1,Role=admin)

[0036] ③ Example of logs from vendor B's switch:

[0037] <190> Dec 8 21:43:57 2002Lyz-CQZ-CSW01-S5500%%10SC / 6 / SC_AAA_LAUNCH(l):-DevIP=10.0.53.31-AAAType=ACCOUNT-AAAScheme=local-Service=login-UserName=admin123@system; AAA launched.

[0038] ④ Example of logs from Vendor C firewall:

[0039] <130> netscreen:NetScreen device_id=netscreen[Root]system-notification-00257(traffic):start_time="2014-04-24 16:01:32"duration=0policy_id=1service=dns proto=17src zone=Trust dst zone=Untrust action=Deny sent=0rcvd=0src=192.168.44.34dst=8.8.8.8src_port=49154dst_port=53session_id=0

[0040] ⑤ Log example from vendor D switch:

[0041] <187> 40282:Apr 13 14:40:49.363:%LINK-3-UPDOWN:InterfaceGigabitEthernet1 / 0 / 13,changed state to down

[0042] As can be seen from the Syslog log examples above, the main message content of Syslog logs mostly uses a key+value format, separated by delimiters. Additionally, some vendors, considering the relatively fixed log format, reduce the log message size by omitting the key and only displaying the value in the log, separated by delimiters, as shown in the log message body example of this application:

[0043] 108|^127.0.0.1|^Smart Monitoring Terminal Name|^123325222|^1|^127.0.0.1|^Firewall|^524|^127.0.0.1|^Audit|^8888|^TCP|^16|^servicenum|^objectnum|^tableservice|^tableioi|^pccccmd|^pcccfnc|^

[0044] In view of the specific structure of Syslog logs mentioned above, this invention provides an automatic Syslog log parsing method. This method obtains the approximate structure of Syslog logs by automatically parsing them from networks or files. For example... Figure 1 As shown, the method specifically includes:

[0045] Step 110: Collect a large number of Syslog log messages from a single source;

[0046] Specifically, a large number of Syslog log messages are collected from a single device, ensuring a single source of data. The log format is standardized for each device to ensure diverse message content for comprehensive analysis. For example, a program on a PC receives Syslog logs and saves them to a database or file.

[0047] Step 120: Remove the header information from the Syslog log, retain the message body content, and parse it into a string collection;

[0048] Since the header information of Syslog logs is in a fixed format and is relatively simple, this application does not consider parsing the header information when analyzing Syslog logs, but only parses the message body content, which is then parsed into a set of strings.

[0049] Step 130: Analyze the string set and determine the delimiter;

[0050] The delimiter set consists of visible characters on the keyboard that do not contain letters, excluding some special characters, mainly including @='”?.: / \. These characters are frequently used in the content and therefore will not be used as delimiters between fields. The remaining character set, i.e., the delimiters, consists of spaces and ~`!#$%^&*_-+|;,()[]{}<>.

[0051] The algorithm for confirming the delimiter is as follows:

[0052] For the set of delimiters listed in this application, a frequent itemset algorithm (such as the Apriori algorithm) is used to find frequent sequences of delimiters. These sequences may be single characters or sets of multiple characters. Since the delimiter set is fixed, totaling more than 20 characters, the algorithm for finding frequent itemsets is highly efficient and suitable for practical engineering applications. Because the log delimiter for the same device from the same manufacturer is always fixed and appears multiple times, and the delimiter must come from the previously given specific character set, the results obtained by this algorithm are relatively reliable. If the algorithm's calculation results in character combinations, these combinations are used preferentially.

[0053] Step 140: Use the defined delimiter to split the Syslog log message body content, and split all logs into string arrays according to the delimiter;

[0054] Step 150: Confirm whether the string "key" exists in the string array;

[0055] The key is a phrase or combination that appears in every log entry, and the key and value are usually connected by special symbols, typically an equals sign or a colon. The algorithm for confirming the existence of a key in a string array is to check if every string in the array contains an equals sign or a colon.

[0056] Step 160: Classify the strings according to whether a key value exists. If a key exists, classify the logs according to the key and data length. If no key exists, check the data type and classify the logs according to the array length and data type.

[0057] Because the arrays after splitting each type of log have the same length and format, they can be easily divided into different groups. The classification algorithm is as follows: classify by array length, ensuring that logs of the same type have the same length after splitting; for arrays of the same length, check the similarity of each item; for logs with keys, directly check the consistency of keys, as logs of the same type have the same keys; for logs without keys, check the consistency of data types, as logs of the same type have consistent data types in the same position, mainly checking for numbers, IP addresses, dates (times), and strings.

[0058] Step 170: Automatically generate parsing templates for different categories of logs, use the parsing templates to extract the value corresponding to each field, and perform subsequent field mapping or transformation normalization processing.

[0059] After classifying the strings, each category has a consistent log format and clearly defined fields. Therefore, based on the presence or absence of a key, a parsing template (i.e., a regular expression) is automatically generated for that category of logs, allowing the extraction of the corresponding field's value. For logs with a key, the key and value can be separated, and the user performs field mapping and normalization. For logs without a key, the value is separated into a list of values, and the user adds the key based on the log's documentation, followed by normalization.

[0060] This invention fully utilizes the characteristics of the Syslog log protocol, automatically identifies delimiters, leverages the known features of the Syslog protocol, performs analysis based on these features, and automatically constructs log parsing templates, making it relatively convenient to model and analyze Syslog logs.

[0061] Corresponding to the above embodiments, the present invention provides a computer storage medium, including: at least one memory and at least one processor;

[0062] The memory is used to store one or more program instructions;

[0063] A processor is used to run one or more program instructions to execute a Syslog log automatic parsing method.

[0064] Corresponding to the above embodiments, this embodiment of the invention provides a computer-readable storage medium containing one or more program instructions, which are executed by a processor to provide an automatic Syslog log parsing method.

[0065] The embodiments disclosed in this invention provide a computer-readable storage medium storing computer program instructions that, when executed on a computer, cause the computer to perform the aforementioned Syslog log automatic parsing method.

[0066] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0067] The various methods, steps, and logic diagrams disclosed in the embodiments of this invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods.

[0068] The storage medium can be memory, such as volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0069] Among them, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.

[0070] Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0071] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0072] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using a combination of hardware and software. When applied as software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0073] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatic parsing of Syslog logs, characterized in that, The method comprises the following steps: Collecting a large number of single-source Syslog log messages; Removing the header information of the Syslog log and retaining the message body content, and parsing the message body content into a string set; Analyzing the string set to determine the delimiter; Segmenting the Syslog log message body content using the determined delimiter, and dividing all logs into a string array according to the delimiter; Confirming whether a key exists in the string in the string array; Classifying the string according to whether the key value exists, if the key exists, classifying the log according to the key and the data length, if the key does not exist, checking the data type of the data, and classifying the log according to the array length and the data type; Automatically generating a parsing template for different classified logs, using the parsing template to extract the value corresponding to each field, and performing subsequent field mapping or conversion normalization processing Because the array length after segmentation of each type of log is the same and the format is consistent, the classification algorithm is as follows: classifying according to the array length, the length of the same type of log after segmentation is the same; checking the similarity of each item for the same length array; for the log with the key, checking the consistency of the key, the key of the same type of log is the same; for the log without the key, checking the consistency of the data type, the data type at the same position of the same type of log is consistent, checking the number, IP address, date, and string data types.

2. The Syslog log automatic parsing method of claim 1, wherein, A program on a PC receives the Syslog log, and saves the Syslog log to a database or a file.

3. The Syslog log automatic parsing method of claim 1, wherein, The delimiter set includes spaces and ~`!#$%^&*_-+|;,()[]{}<>.

4. The Syslog log automatic parsing method of claim 1, wherein, For the delimiter character set listed in the application, a frequent item set algorithm is used to find the frequent sequence of the segmentation character, which is a single character or a set of multiple characters.

5. The Syslog log automatic parsing method of claim 1, wherein, The algorithm for confirming whether a key exists in the string array is: checking whether an equal sign or a colon is included in each string in the string array.

6. A computer storage medium, characterized in that The method comprises the following steps: At least one memory and at least one processor; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the Syslog log automatic parsing method according to any one of claims 1-5.

Images