Matching optimization method and system for security log analysis rules

Abstract

The invention provides a matching optimization method and system for security log analysis rules, and the method comprises the steps: reading analysis rules in a rule base, extracting keywords of the analysis rules one by one, and associating the keywords with the corresponding analysis rules; according to the method, the content of the security log is matched with the keywords, the analysis rules corresponding to the successfully matched keywords serve as the rule set to be used for analyzing the security log, the keywords in the analysis rules are extracted, and the keywords are associated with the corresponding analysis rules, so that the problems of multiple matching times, large resource consumption and the like are solved; according to the method, firstly, the content of the security log is matched with the keyword, the analysis rule corresponding to the successfully matched keyword is screened out, and only the analysis rule needs to be traversed, that is, linear traversal is converted into a tree structure, rapid matching is completed, and the efficiency is greatly improved.

Description

Technical Field

[0001] This application relates to the field of network security technology, and in particular to a method and system for optimizing the matching of security log parsing rules. Background Technology

[0002] With the deepening of digital transformation, network architecture continues to expand, and business types and application scenarios are becoming increasingly complex. Enterprises and security operations centers are generating massive amounts of log data from their IT infrastructure, business systems, and security protection devices. These logs need to be parsed line by line according to rules, and each log entry needs to be matched against a large number of rules, resulting in high CPU consumption and long response times, making it difficult to meet the processing needs of massive log volumes. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for optimizing the matching of security log parsing rules, which filters rules from the rule base, reduces the number of matching operations, reduces resource consumption, and improves parsing speed.

[0004] To address the aforementioned problems, a first aspect of the present invention provides a method for optimizing the matching of security log parsing rules, comprising: Step S1: Read the parsing rules in the rule base, extract the keywords of each parsing rule, and associate the keywords with the corresponding parsing rules; Step S2: Match the content of the security log with the keyword, and use the parsing rule corresponding to the successfully matched keyword as a rule set to parse the security log.

[0005] Furthermore, in the above-mentioned method for optimizing the matching of security log parsing rules, after the keywords in step S1 are associated with the corresponding parsing rules, a multi-pattern matching tree is constructed based on the keywords.

[0006] Furthermore, in the above-mentioned method for optimizing the matching of security log parsing rules, the content of the security log in step S2 is matched with the keyword in the multi-pattern matching tree constructed by the keyword.

[0007] Furthermore, step S1 in the above-mentioned method for optimizing the matching of security log parsing rules includes: Step S11: Divide the parsing rules containing the filtering conditions into first rules, and the rest into second rules, wherein the filtering conditions include the keywords which are manually defined; Step S12: Extract the manually defined keywords from the first rule one by one, associate them with the corresponding first rule, and construct the first multi-pattern matching tree based on the keywords; Step S13: Extract the keywords from the expressions in the second rule one by one, associate them with the corresponding second rule, and construct a second multi-pattern matching tree based on the keywords; Step S2 includes: Step S21: The content of the security log is preferentially matched with the first multi-pattern matching tree to obtain the first rule set; Step S22: If step S21 fails to match, then match with the second multi-pattern matching tree to obtain the second rule set.

[0008] Furthermore, the method for optimizing the matching of security log parsing rules, as described above, includes the following after step S22: Step S23: If step S22 cannot be matched, the parsing rules in the rule base that are not included in the first rule set and the second rule set are used as the third rule set, and the security log is matched with the third rule set to obtain the third rule set.

[0009] Furthermore, the method for optimizing the matching of security log parsing rules, as described above, includes the following after step S2: Step S3: Associate the security log with the rule set to obtain the association result as a candidate rule set. In the next matching, the candidate rule set will be matched with the security log first. If the content of the security log in the next matching is the same or similar to that of the security log in the candidate rule set, the corresponding parsing rule will be retrieved from the rule set.

[0010] Furthermore, the keyword cleaning, deduplication, and normalization processing in step S12 of the above-mentioned security log parsing rule matching optimization method.

[0011] Furthermore, in the above-mentioned method for optimizing the matching of security log parsing rules, when the content of the security log is matched with the keyword in step S2, case sensitivity is ignored.

[0012] According to another aspect of the present invention, the present invention also provides a matching optimization system for security log parsing rules, comprising: The keyword extraction module is used to read the parsing rules in the rule base, extract the keywords of the parsing rules one by one, and associate the keywords with the corresponding parsing rules; The rule filtering module is used to match the content of the security log with the keywords, and to use the parsing rules corresponding to the successfully matched keywords as a rule set for parsing the security log.

[0013] The above-mentioned technical solution of the present invention has the following beneficial technical effects: by extracting keywords from the parsing rules and associating the keywords with the corresponding parsing rules, when it is necessary to parse the security log, the content of the security log is first matched with the keywords, and the parsing rules corresponding to the successfully matched keywords are filtered out. Only this part of the parsing rules needs to be traversed, that is, the linear traversal is transformed into a tree structure, which completes the fast matching and greatly improves efficiency. Attached Figure Description

[0014] Figure 1 This is a flowchart of the steps according to an embodiment of the present invention; Figure 2 This is a structural block diagram according to an embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the present invention. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concept of the present invention. In the description of the present invention, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0016] refer to Figure 1 The following is an embodiment of the matching optimization method for security log parsing rules in this invention, including: Step S1: Read the parsing rules in the rule base, extract the keywords of each parsing rule, and associate the keywords with the corresponding parsing rules; In step S1, after the keyword is associated with the corresponding parsing rule, a multi-pattern matching tree is constructed based on the keyword. Specifically, step S1 includes: Step S11: Divide the parsing rules containing the filtering conditions into first rules, and the rest into second rules, wherein the filtering conditions include the keywords which are manually defined; Step S12: Extract the manually defined keywords from the first rule one by one, associate them with the corresponding first rule, and construct the first multi-pattern matching tree based on the keywords; The keyword cleaning, deduplication, and normalization processes in step S12; Step S13: Extract the keywords from the expressions in the second rule one by one, associate them with the corresponding second rule, and construct a second multi-pattern matching tree based on the keywords; Step S2: Match the content of the security log with the keywords, and use the parsing rules corresponding to the successfully matched keywords as a rule set to parse the security log; When the content of the security log in step S2 matches the keyword, case sensitivity is ignored. In step S2, the content of the security log is matched with the keyword in the multi-pattern matching tree constructed using the keyword. Step S2 includes: Step S21: The content of the security log is preferentially matched with the first multi-pattern matching tree to obtain the first rule set; Step S22: If step S21 fails to match, then match with the second multi-pattern matching tree to obtain the second rule set; Step S23: If step S22 cannot be matched, the parsing rules in the rule base that are not included in the first rule set and the second rule set are used as the third rule set, and the security log is matched with the third rule set to obtain the third rule set; Step S3: Associate the security log with the rule set to obtain the association result as a candidate rule set. In the next matching, the candidate rule set will be matched with the security log first. If the content of the security log in the next matching is the same or similar to that of the security log in the candidate rule set, the corresponding parsing rule will be retrieved from the rule set.

[0017] The rule base for security logs contains a vast number of parsing rules. Parsing security logs requires traversing all rules one by one, a method that consumes significant resources and is extremely slow. Existing systems often have parsing rules that simultaneously include multiple pieces of information: filtering conditions for coarse-grained selection, expression logic for field extraction and transformation, rule priorities, etc. This information is stored mixed in the same data structure, requiring a traversal of each rule, resulting in low matching efficiency. Therefore, it is necessary to select useful parsing rules from the massive rule base to improve efficiency. In this embodiment, keywords are first extracted from the parsing rules, and then associated with the corresponding parsing rules, i.e., the rule ID of the parsing rule is recorded, and the keywords are bound to their corresponding rule IDs. When parsing security logs, the content of the security logs is first matched against the keywords, filtering out the parsing rules corresponding to the successfully matching keywords. Only these matching rules need to be traversed, transforming the linear traversal into a tree structure for rapid matching and significantly improving efficiency.

[0018] During long-term system operation, parsing rules formed by keywords manually input by analysts are often more accurate, such as security log source, machine code, and source address combination identifiers. Analysts can also determine parsing rules based on past experience; these rules have higher priority. Therefore, when parsing security logs, the parsing rules in the rule base can be categorized first, and the higher-priority rules can be filtered out. Thus, the parsing rules are divided into first rules and second rules, with the first rule having higher priority. Keywords for the first and second rules are extracted separately, and the keywords are associated with their corresponding parsing rules. Many parsing rules use regular expressions. When extracting these types of parsing rules, consecutive alphanumeric characters, underscores, and hyphen literals are identified as keywords in the text. Escape characters, groupings, character classes, or metacharacters are truncated. For example, the expression is: (alarm).*?risktype":"(.*?)","risksrc":"(.*?)".+handled":"(.+?)","level":"(.+?)","description":"(.+?)",.+?"ip":"(\d+\.\d+\.\d+\.\d+)".*?"comment":"(.*?)".

[0019] Therefore, the extracted keywords are: alarm, risktype, level, handled, description, comment, risksrc. If it is a JSON type, then the keywords are extracted from the keys at each level of the path in the JSON PATH of the expression.

[0020] When new security logs enter the system, to improve matching efficiency, multi-pattern matching technology can be used, i.e., constructing a multi-pattern matching tree. Therefore, a first multi-pattern matching tree and a second multi-pattern matching tree are constructed. First, the security log is matched against the first multi-pattern matching tree. If a match is successful, the parsing rules corresponding to the successfully matched keywords are selected as a rule set for parsing the security log. If a match fails, the system continues matching against the second multi-pattern matching tree. If a match is successful, the parsing rules corresponding to the successfully matched keywords are selected as a rule set for parsing the security log. If a match fails, the remaining parsing rules are traversed, i.e., rules not included in the first or second rule sets. This step serves as a fallback strategy and can be skipped or optional.

[0021] After each matching of security logs and parsing rules is completed, the security logs are associated with the matching rule set as a candidate rule set. In the next matching, if the content of the security logs is the same or similar to that of the security logs in the rule set, the corresponding parsing rule is retrieved from the rule set, thereby avoiding duplicate calculations.

[0022] Secondly, refer to Figure 2 The present invention also provides a matching optimization system for security log parsing rules, comprising: The keyword extraction module is used to read the parsing rules in the rule base, extract the keywords of the parsing rules one by one, and associate the keywords with the corresponding parsing rules; The rule filtering module is used to match the content of the security log with the keywords, and to use the parsing rules corresponding to the successfully matched keywords as a rule set for parsing the security log.

[0023] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A method for optimizing the matching of security log parsing rules, characterized in that, include: Step S1: Read the parsing rules in the rule base, extract the keywords of each parsing rule, and associate the keywords with the corresponding parsing rules; Step S2: Match the content of the security log with the keyword, and use the parsing rule corresponding to the successfully matched keyword as a rule set to parse the security log.

2. The matching optimization method for security log parsing rules according to claim 1, characterized in that: In step S1, after the keywords are associated with the corresponding parsing rules, a multi-pattern matching tree is constructed based on the keywords.

3. The matching optimization method for security log parsing rules according to claim 2, characterized in that: In step S2, the content of the security log is matched with the keyword in the multi-pattern matching tree constructed using the keyword.

4. The matching optimization method for security log parsing rules according to claim 3, characterized in that: Step S1 includes: Step S11: Divide the parsing rules containing the filtering conditions into first rules, and the rest into second rules, wherein the filtering conditions include the keywords which are manually defined; Step S12: Extract the manually defined keywords from the first rule one by one, associate them with the corresponding first rule, and construct the first multi-pattern matching tree based on the keywords; Step S13: Extract the keywords from the expressions in the second rule one by one, associate them with the corresponding second rule, and construct a second multi-pattern matching tree based on the keywords; Step S2 includes: Step S21: The content of the security log is preferentially matched with the first multi-pattern matching tree to obtain the first rule set; Step S22: If step S21 fails to match, then match with the second multi-pattern matching tree to obtain the second rule set.

5. The matching optimization method for security log parsing rules according to claim 4, characterized in that: Following step S22, the following is also included: Step S23: If step S22 cannot be matched, the parsing rules in the rule base that are not included in the first rule set and the second rule set are used as the third rule set, and the security log is matched with the third rule set to obtain the third rule set.

6. The matching optimization method for security log parsing rules according to claim 4, characterized in that: The process following step S2 also includes: Step S3: Associate the security log with the rule set to obtain the association result as a candidate rule set. In the next matching, the candidate rule set will be matched with the security log first. If the content of the security log in the next matching is the same or similar to that of the security log in the candidate rule set, the corresponding parsing rule will be retrieved from the rule set.

7. The matching optimization method for security log parsing rules according to claim 4, characterized in that: The keyword cleaning, deduplication, and normalization processes in step S12.

8. The matching optimization method for security log parsing rules according to claim 4, characterized in that: When the content of the security log in step S2 matches the keyword, case sensitivity is ignored.

9. A system for matching and optimizing security log parsing rules, characterized in that, include: The keyword extraction module is used to read the parsing rules in the rule base, extract the keywords of the parsing rules one by one, and associate the keywords with the corresponding parsing rules; The rule filtering module is used to match the content of the security log with the keywords, and to use the parsing rules corresponding to the successfully matched keywords as a rule set for parsing the security log.

Images