Log parsing method and system

Abstract

The present invention discloses a log parsing method and system, the method comprising the following steps: creating a parsing rule base, wherein the parsing rule base stores a comparison table of log content and parsing files; setting log preprocessing rules, generating a preprocessing rule word base according to the original log content, and generating preprocessing rules based on the preprocessing rule word base; obtaining a log to be parsed, processing the log to be parsed using the preprocessing rule word base, and obtaining the type of the log to be parsed; dividing different types of logs to be parsed into a plurality of different independent lines, and parsing the logs using parsing files; and correcting and updating the preprocessing rule word base using the parsed content. The present invention solves the problems of poor parsing versatility, slow parsing speed, and low efficiency in log parsing.

Description

Technical Field

[0001] The present invention relates to the field of log analysis, and in particular to a log analysis method and system. Background Art

[0002] Connected devices generate a large amount of log data every day. The log data contains important information such as the operating status of hardware and applications, error information, user behavior, etc. However, the log formats generated by various devices are usually different. As a result, the information between logs cannot be unified and the data cannot be effectively obtained. Therefore, it is necessary to parse the logs and parse the logs of different formats into structured data to facilitate data statistics and analysis.

[0003] The parsed log content can help enterprises and organizations better understand the operation of their hardware devices and applications, and based on the unified specification content, specific logs can be processed uniformly so that users can quickly perceive the key content of the log content, thereby improving the reliability, performance and security of the system and equipment.

[0004] Currently, the following solutions are mainly used to parse and process logs:

[0005] A log parsing file is written in combination with the device log format description file and the log content / format fields. The file contains the correspondence between the original log fields and the standard fields defined by the log platform. After binding the parsing file to the log sending source IP, the parsing file of the corresponding log format is matched in the order of log reception time. The original information in the log is parsed into the fields standardized by the log platform, and then the log content after parsing is analyzed to extract key content to generate alarms or perform other processing.

[0006] The benefit of this solution is that it can guarantee the reliability of the log analysis data source, ensuring that the access and analysis of each log is within the plan. At the same time, the logical structure of log analysis is relatively simple, but the problems it brings include:

[0007] (1) Each time you connect to a device, you need to understand its log format and parsing rules in advance, so that you can configure the corresponding parsing file for log parsing. Otherwise, the log cannot be parsed.

[0008] (2) If the number of logs sent within a certain period of time is large, the log parsing speed will not be able to keep up with the log sending speed, resulting in log accumulation, which will affect the overall parsing and receiving speed of logs;

[0009] (3) All logs must be parsed before key content can be extracted. However, the parsing order will result in some abnormal information not being immediately known.

[0010] (4) During the analysis process, the logical relationship between the data is close, and if an error occurs in any link, it will cause deviations in the operation of the overall system. Summary of the invention

[0011] In order to solve the technical problems of poor parsing versatility, slow parsing speed and low efficiency in current log parsing, the present invention proposes a log parsing method and system, which includes the following steps:

[0012] S1. Create a parsing rule base, in which a comparison table of log content and parsing files is stored;

[0013] S2. Set log preprocessing rules and generate a preprocessing rule vocabulary based on the original log content;

[0014] S3, obtaining the log to be parsed, and processing the log to be parsed using the preprocessing rule vocabulary to obtain the type of the log to be parsed;

[0015] S4, dividing different types of logs to be parsed into multiple independent lines, and using the parsing file to perform log parsing;

[0016] S5. Correct and update the pre-processing rule vocabulary using the parsed content.

[0017] A log parsing system, comprising:

[0018] Parsing rule building module: creating a parsing rule base, in which a comparison table of log content and parsing files is stored;

[0019] Preprocessing rule setting module: set log preprocessing rules and generate preprocessing rule vocabulary according to the original log content;

[0020] Log classification module: obtains the log to be parsed, processes the log to be parsed using the preprocessing rule vocabulary, and obtains the type of the log to be parsed;

[0021] Log parsing module: divides different types of logs to be parsed into multiple independent lines, and uses parsing files to perform log parsing;

[0022] Optimization and updating module: using the parsed content to correct and update the pre-processing rule vocabulary.

[0023] The beneficial effects provided by the present invention are:

[0024] 1. Security monitoring: The log information of various devices will contain event information. Based on the sent log information, the security events of the system can be monitored, and security threats can be discovered and responded to in a timely manner to prevent key logs from being submerged in massive log information, leading to larger accidents. At the same time, all types of logs can be analyzed.

[0025] 2. Record equipment operation information. After storing and parsing the logs of each device, not only can the fault be quickly located when a device fails, but the system performance and response time can also be optimized in daily work.

[0026] 3. Data analysis: By parsing a large amount of data from logs, we can analyze the patterns and trends in the current data, improve the efficiency of analysis, and at the same time grasp the level of network system construction and security management capability construction.

[0027] 4. Optimize resource utilization. Through the operation of the equipment, you can understand the utilization of system resources, optimize resource allocation and utilization, and improve system utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic flow chart of the method of the present invention;

[0029] Figure 2 The figure is a detailed schematic diagram of the process of processing a log by the method of the present invention. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0031] Please refer to Figure 1 , Figure 1 It is a schematic flow chart of the method of the present invention.

[0032] The present invention provides a log parsing method and system, including:

[0033] The method comprises the following steps:

[0034] S1. Create a parsing rule base, in which a comparison table of log content and parsing files is stored;

[0035] It should be noted that step S1 is specifically as follows:

[0036] Obtain historical parsing files, summarize specific values ​​of a certain type of parsing files, and obtain key parameters;

[0037] For logs sent by a previously unbound log IP source, the original content of the log is matched with the key parameters. If the match is successful, it means that the log corresponds to the type of parsed file, forming a comparison table.

[0038] As an embodiment, the present invention summarizes the specific values ​​of the parsing file to form some key parameters. For the logs sent by the log source IP that is not bound to the parsing file, the original content of the log is matched with the key parameters of the parsing file. After the matching is completed, the corresponding parsing file is used for parsing.

[0039] Specifically, the specific value refers to the content of a value in the original log that is parsed in the expression in the parsing file. For example, the alarm level in the original log is expressed as level l, and after parsing the parsing file, level l will be converted into the alarm level.

[0040] The value of the level field in the original log may be 1, 2, 3, or 4, which needs to be mapped to the alarm level in the platform in the parsed file.

[0041] Therefore, level 4 is a group of keywords. When the original log contains this content, it is parsed using the parsing file corresponding to the keywords.

[0042] S2. Set log preprocessing rules and generate a preprocessing rule vocabulary based on the original log content;

[0043] It should be noted that step S2 specifically includes: extracting key parts from the original content of the log as the preprocessing rule vocabulary.

[0044] As an embodiment, the present invention extracts relatively critical content from the original log information and summarizes it into a preprocessing rule vocabulary, and then generates preprocessing rules based on the AND or NOT relationship between the rule vocabulary content and the log source.

[0045] The specific process of forming preprocessing rules is as follows:

[0046] 1. First, add keywords as conditions for the preprocessing rules to form a vocabulary. The added keywords are the content that may appear in the original log information. At the same time, you can set the rules under which the keywords are allowed to be cited. The conditions include: log source IP

[0047] 2. Create pre-processing rules: Create pre-set rules based on the content of the added keyword library and the accompanying information of the log transmission. Users can select multiple types of information for "AND or NOT" combination. The specific instructions are as follows:

[0048] Please refer to Table 1 and Table 2. Table 1 is the accompanying information of log transmission, and Table 2 is the description of the information combination method.

[0049] Table 1 Additional information for log transmission

[0050]

[0051]

[0052] Table 2 Description of information combination method

[0053]

[0054] 3. Set the execution scope / conditions for the pre-processing rules. Set the scope of rule execution by combining the above-mentioned log transmission incidental information. When the incidental information of the received log meets the execution conditions set for a certain rule, execute this rule.

[0055] 4. Set the execution mode after the current log meets the set preprocessing rules. There are two types: 1. Mark this log as an alarm log or other special event to enter the parsing stage; 2. Directly discard this log without parsing.

[0056] S3, obtaining the log to be parsed, and processing the log to be parsed using the preprocessing rule vocabulary to obtain the type of the log to be parsed;

[0057] It should be noted that the different types of logs to be parsed include normal logs and alarm logs. Before the log is parsed, the pre-processing rules are executed on its original content, and the log is discarded and not parsed or marked as an alarm log and enters the alarm parsing stage (not in the same line as the normal log parsing) according to the processing method set by its rules.

[0058] S4, dividing different types of logs to be parsed into multiple independent lines, and using the parsing file to perform log parsing;

[0059] It should be noted that for normally parsed logs, after parsing is completed, the parsed log contents are analyzed to extract key information and generate alarms or perform other processing based on the analysis rules.

[0060] S5. Correct and update the pre-processing rule vocabulary using the parsed content.

[0061] It should be noted that, for normal logs, after they are parsed, key information is extracted from the parsed content, and the key information is used as the alarm judgment standard when parsing the alarm log.

[0062] Based on the content generated after normal log analysis, if the content is finally identified as alarm content, the field content of the normal log triggering the preprocessing rule is used to generate preprocessing keywords in the value of the original log and fill the preprocessing rule vocabulary.

[0063] After log parsing is completed, the preprocessing keywords of the logs marked as alarms that have been preprocessed and not matched with the analysis rules are removed from the preprocessing rule vocabulary.

[0064] As an example, please refer to Figure 2 , Figure 2 This is the detailed process of processing a new log.

[0065] 1. When the log platform receives a new log, it will store the log and trigger different processing paths based on the attached attributes of the sent log and its original content, so as to achieve the purpose of step-by-step analysis of different types of logs to quickly understand the log information content and process it.

[0066] 2. The specific process is as follows: after receiving the log, first determine whether the parsing file is bound according to the source IP address of the log. If bound, parse it with the corresponding parsing file. If not bound, match the original log content with the specific values ​​of all parsing files. If the match is successful, parse the log with the parsing file corresponding to the specific value. If not, do not parse the log. When all logs are associated with the parsing file, enter the log preprocessing phase. Filter the logs according to the execution range set by the rules to see if preprocessing is required. If preprocessing is not required, directly enter the normal log parsing channel for log parsing (in chronological order). If the log needs to execute the preprocessing rules, when the log does not meet the preprocessing rules, it also enters the normal parsing channel. If it meets the rules, it is processed differently according to the execution method of the rules: 1. Abandon the log directly without parsing. 2. Mark the log as an alarm log and enter the log parsing phase, which is not in the same parsing channel (alarm parsing channel) as the normal log. When the log parsing through the normal parsing channel is completed, in addition to being stored in the database, it also needs to be matched with the parsed log alarm rules. If the rules are met, an alarm log will also be generated. At the same time, after the log alarm is generated, the original content of the log information content of the matching parsed log alarm rules before parsing will be filled into the preprocessing keyword library.

[0067] A log parsing system, comprising:

[0068] Parsing rule building module: creating a parsing rule base, in which a comparison table of log content and parsing files is stored;

[0069] Preprocessing rule setting module: set log preprocessing rules and generate preprocessing rule vocabulary according to the original log content;

[0070] Log classification module: obtains the log to be parsed, processes the log to be parsed using the preprocessing rule vocabulary, and obtains the type of the log to be parsed;

[0071] Log parsing module: divides different types of logs to be parsed into multiple independent lines, and uses parsing files to perform log parsing;

[0072] Optimization and updating module: using the parsed content to correct and update the pre-processing rule vocabulary.

[0073] Finally, the processing of the entire method is described through an embodiment.

[0074] For example, a normal log is: level 5 Fr i Ju l 21 2023 09:32:04 192.168.184.145 Kelai Network Full Traffic Security Analysis System - Server: Trigger time: 2023-07-21 09:32:04 Link name: Fenghuo Innovation Valley Alarm name: A data flow characteristic value alarm Alarm level: High Alarm type: Information collection Trigger condition: HTTPOPTIONS method Alarm type: Data flow characteristic value alarm Source IP address: 31.208.190.177 Source port: 25074 Target IP address: 192.168.201.211 Target port: 4413 Cumulative trigger times: 1;

[0075] An alarm log is level3 Fr i Ju l 21 2023 09:32:04 192.168.184.145 Kelai Network Full Traffic Security Analysis System - Server: Trigger time: 2023-07-21 09:32:04 Link name: Fenghuo Innovation Valley Alarm name: A data flow characteristic value alarm Alarm level: High Alarm type: Information collection Trigger condition: HTTPOPTIONS method Alarm type: Data flow characteristic value alarm Source IP address: 31.208.190.177 Source port: 25074 Target IP address: 192.168.201.211 Target port: 4413 Cumulative trigger times: 1;

[0076] The specific value of the parsed file is: Kelai Network Full Traffic Security Analysis System

[0077] The preprocessing rule base is level 3 and level 4; the preprocessing rule is: Enter the alarm log parsing channel

[0078] The parsed content is; Log level: Advanced Log receiving time: 2023 / 12 / 02 20:13:25 Field 1:......................

[0079] After parsing, the updated preprocessing rules are: level 3, level 4, level 5.

[0080] The beneficial effects of the present invention are:

[0081] 1. Security monitoring: The log information of various devices will contain event information. Based on the sent log information, the security events of the system can be monitored, and security threats can be discovered and responded to in a timely manner to prevent key logs from being submerged in massive log information, leading to larger accidents. At the same time, all types of logs can be analyzed.

[0082] 2. Record equipment operation information. After storing and parsing the logs of each device, not only can the fault be quickly located when a device fails, but the system performance and response time can also be optimized in daily work.

[0083] 3. Data analysis: By parsing a large amount of data from logs, we can analyze the patterns and trends in the current data, improve the efficiency of analysis, and at the same time grasp the level of network system construction and security management capability construction.

[0084] 4. Optimize resource utilization. Through the operation of the equipment, you can understand the utilization of system resources, optimize resource allocation and utilization, and improve system utilization.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log parsing method, characterized in that: The following steps are involved: S1. Create a parsing rule base, in which a comparison table of log content and parsing files is stored; S2. Set log preprocessing rules, generate a preprocessing rule vocabulary according to the original content of the log, and generate preprocessing rules based on the preprocessing rule vocabulary; The specific process of forming preprocessing rules is as follows:

1. Create preset rules based on the content of the added keyword library and the accompanying information of the log transmission; 2. Create preset rules based on the content of the added keyword library and the accompanying information of log transmission. You can select multiple types of information to combine "AND" or "NOT"; 3. Set the execution scope / condition for the pre-processing rule. The execution scope of the rule is set by combining the above-mentioned log transmission incidental information. When the incidental information of the received log meets the execution condition set for a certain rule, the rule is executed.

4. Set the execution mode after the current log meets the set pre-processing rules. There are two types: mark this log as an alarm log or other special event to enter the parsing stage, or directly discard this log without parsing; S3, obtaining the log to be parsed, and processing the log to be parsed using the preprocessing rule vocabulary to obtain the type of the log to be parsed; S4, dividing different types of logs to be parsed into multiple independent lines, and using the parsing file to perform log parsing; S5. Correct and update the pre-processing rule vocabulary using the parsed content.

2. A log parsing method as claimed in claim 1, characterized in that: Step S1 is specifically as follows: Obtain historical parsing files, summarize specific values ​​of a certain type of parsing files, and obtain key parameters; For logs sent by a previously unbound log IP source, the original content of the log is matched with the key parameters. If the match is successful, it means that the log corresponds to the type of parsed file, forming a comparison table.

3. A log parsing method as claimed in claim 1, characterized in that: Step S2 specifically includes: extracting key parts from the original content of the log as a preprocessing rule vocabulary.

4. A log parsing method as claimed in claim 1, characterized in that: The different types of logs to be parsed include: normal logs and alarm logs.

5. A log parsing method as claimed in claim 1, characterized in that: For normal logs, after they are parsed, key information is extracted from the parsed content, and the key information is used as the alarm judgment standard when parsing the alarm log.

6. A log parsing method as claimed in claim 1, characterized in that: The specific process of correcting and updating the preprocessing rule vocabulary in step S5 is: based on the content generated after normal log analysis, if the content is finally identified as alarm content, the field content of the normal log triggering the preprocessing rule is used to generate preprocessing keywords in the value of the original log, and the preprocessing rule vocabulary is filled.

7. A log parsing method as claimed in claim 1, characterized in that: The specific process of correcting and updating the preprocessing rule word library in step S5 also includes: removing the preprocessing keywords of the log marked as an alarm after preprocessing from the preprocessing rule word library that are not matched to the analysis rule after the log parsing is completed.

8. A log parsing system, characterized in that: include: Parsing rule building module: creating a parsing rule base, in which a comparison table of log content and parsing files is stored; Preprocessing rule setting module: set log preprocessing rules and generate preprocessing rule vocabulary according to the original log content; The specific process of forming preprocessing rules is as follows:

1. Create preset rules based on the content of the added keyword library and the accompanying information of the log transmission; 2. Create preset rules based on the content of the added keyword library and the accompanying information of log transmission. You can select multiple types of information to combine "AND" or "NOT"; 3. Set the execution scope / condition for the pre-processing rule. The execution scope of the rule is set by combining the above-mentioned log transmission incidental information. When the incidental information of the received log meets the execution condition set for a certain rule, the rule is executed.

4. Set the execution mode after the current log meets the set pre-processing rules. There are two types: mark this log as an alarm log or other special event to enter the parsing stage, or directly discard this log without parsing; Log classification module: obtains the log to be parsed, processes the log to be parsed using the preprocessing rule vocabulary, and obtains the type of the log to be parsed; Log parsing module: divides different types of logs to be parsed into multiple independent lines, and uses parsing files to perform log parsing; Optimization and updating module: using the parsed content to correct and update the pre-processing rule vocabulary.

Images