Log audit unknown type log classification configuration method

Abstract

The invention relates to the technical field of information security, and provides a log auditing unknown type log classification configuration method which comprises the following steps: step 1, inputting an API token and a basic link of an AI service through an AI configuration page, and storing data into a database; 2, inputting an original log sample to a log classification management page, and triggering an AI interaction module to generate a regular expression of a naming capture group; 3, calling an AI service through an HTTP request, converting an original log into a regular expression with a named capture group according to an instruction, and analyzing a return result; and 4, associating the generated regular expression with the custom log type, and storing the regular expression and the custom log type in a database. By integrating the AI model, a user can automatically generate a high-precision naming capture group regular expression without mastering regular grammar, the technical threshold is reduced, in addition, real-time processing of unknown log types is supported, the classification accuracy is improved to 95% or above, and the response time is smaller than or equal to 500 ms.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to a log audit unknown type log classification configuration method. Background Art

[0002] Log auditing is a core technology in the field of information security, used to collect, store, and analyze various system logs, which is based on manual configuration of regular expressions. Users need to write named capture group regular expressions to associate original logs with custom log types. For example, key fields (such as timestamps and IP addresses) are extracted through capture groups and mapped to display fields, and fixed template matching is also used. The system predefines a limited number of log format templates and achieves classification through pattern matching, but it cannot adapt to new log types. However, with the diversification of log types, traditional log auditing systems have difficulty adapting to logs of unknown types, resulting in a large number of logs that cannot be effectively classified, affecting audit efficiency and accuracy. Summary of the Invention

[0003] In response to the shortcomings of the existing technology, the present invention provides a method for configuring the classification of unknown types of logs in log auditing, which solves the problem that with the diversification of log types, traditional log auditing systems are difficult to adapt to unknown types of logs, resulting in a large number of logs that cannot be effectively classified, affecting audit efficiency and accuracy.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions: A log audit unknown type log classification configuration method, comprising the following steps: Step 1: Enter the API token and base link of the AI ​​service through the AI ​​configuration page and store the data in the database; Step 2: Enter the original log sample into the log category management page to trigger the AI ​​interaction module to generate a named capture group regular expression; Step 3: Call the AI ​​service through HTTP request. The instruction is to convert the original log into a regular expression with named capture groups and parse the returned results. Step 4: Associate the generated regular expression with the custom log type and save it to the database; Step 5: The log receiving module matches the new log according to the classification information and automatically classifies it into the corresponding log type.

[0005] Preferably, the AI ​​service in step 1 is a model based on natural language processing, and a regular expression is generated by the following formula: Regular expression = AI model (original log, instruction) The instruction includes a constraint condition of "removing the capital letter P after the question mark".

[0006] Preferably, the named capture group regular expression generated in step 4 must pass the following verification: Syntax correctness check, using regular expression engine to verify the validity of syntax; Field mapping consistency check to ensure that the capture group names correspond to the log display fields. If the verification fails, an error message is returned and the AI ​​service is requested to generate again.

[0007] Preferably, the association relationship in step 5 supports user-defined adjustment, including: Dynamically add and delete capture group fields; Set the matching threshold, the formula is: .

[0008] Preferably, the AI ​​service is a natural language processing model based on the Transformer architecture, and its regular expression generation response time is ≤300ms and the accuracy rate is ≥98%.

[0009] Preferably, the log receiving module in step 6 adopts a real-time stream processing engine, supports Apache Flink or KafkaStreams, and has a matching delay of ≤10ms.

[0010] Preferably, the AI ​​service supports dynamic switching of multiple services, and the configuration parameters include: AI service provider identification; Service priority weight, calculated as: W = 0.6 × response speed + 0.4 × accuracy.

[0011] A log audit unknown type log classification configuration system includes the following modules: AI configuration module: used to configure API tokens and basic links for AI services and store data in the database; Log classification management module: used to implement regular expression generation, field association and rule storage; Log receiving and matching module, used for log receiving, matching and classification; Database module, used to store API configuration, classification rules and log data; Among them, each module communicates through HTTP protocol or SQL interface.

[0012] Preferably, the log classification management module includes: AI interaction subunit: calls external AI services to generate named capture group regular expressions; Regular expression association subunit: associates regular expressions with log type display fields; Storage mother unit: saves classification rules to the database. The cache module preloads high-frequency regular expressions into the memory.

[0013] Preferably, the log receiving and matching module includes: Log receiving subunit: Receives log data sent from the user host in real time and performs format standardization preprocessing; Real-time matching subunit: Based on the regular expression rules in the database, it matches the logs one by one and extracts the capture group fields; Cache subunit: preloads high-frequency regular expressions into memory, with matching latency ≤ 5ms; Storage subunit: stores the matched log category names and extracted fields into the database; Feedback subunit: Marks unmatched logs as "unknown type," triggers an alarm, and submits it to the AI ​​service to regenerate rules.

[0014] The present invention provides a method for configuring the classification of unknown log types in log auditing. It has the following beneficial effects: 1. By integrating an AI model, the present invention allows users to automatically generate high-precision named capture group regular expressions without having to master regular syntax, thereby lowering the technical threshold. In addition, it supports real-time processing of unknown log types, with classification accuracy increased to over 95% and response time ≤500ms.

[0015] 2. This invention applies natural language processing technology to regular expression generation, breaking through the limitations of traditional reliance on manual labor, achieving full automation from log input to classified storage, improving efficiency by over 70%, and flexibly accessing different AI services through APIs to adapt to various log scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a system diagram of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] Example: Please see the attached Figure 1 The embodiment of the present invention provides a method for configuring the classification of unknown type logs in log audit, including the following steps: Step 1: Enter the API token and base link for the AI ​​service on the AI ​​configuration page and store the data in the database. The AI ​​service is a natural language processing-based model. Generate a regular expression using the following formula: Regular Expression = AI Model (Original Log, Command). The command includes the constraint of "removing the capital letter P after the question mark." Step 2: Enter the original log sample into the log category management page to trigger the AI ​​interaction module to generate a named capture group regular expression; Step 3: Call the AI ​​service through an HTTP request. The instruction is to convert the original log into a regular expression with named capture groups and parse the returned results. The AI ​​service is a natural language processing model based on the Transformer architecture. Its regular expression generation response time is ≤ 300ms and the accuracy is ≥ 98%. The AI ​​service supports dynamic switching between multiple services. Configuration parameters include: AI service provider identification; Service priority weight, calculated as: W = 0.6 × response speed + 0.4 × accuracy; Step 4: Associate the generated regular expression with the custom log type and save it to the database. The generated named capture group regular expression must pass the following verification: Syntax correctness check, using regular expression engine to verify the validity of syntax; Field mapping consistency check to ensure that the capture group names correspond to the log display fields. If the verification fails, an error message is returned and the AI ​​service is requested to generate again; Step 5: Using a real-time stream processing engine, such as Apache Flink or Kafka Streams, the log receiving module with a matching delay of ≤10ms matches new logs based on classification information and automatically categorizes them into the corresponding log type. The association relationships can be customized by the user, including: Dynamically add and delete capture group fields; Set the matching threshold, the formula is: .

[0019] As another aspect of the present invention, an embodiment of the present invention provides a log audit unknown type log classification configuration system, including the following modules: AI configuration module: used to configure API tokens and basic links for AI services and store data in the database; The log classification management module used to implement regular expression generation, field association, and rule storage includes: AI interaction subunit: calls external AI services to generate named capture group regular expressions; Regular expression association subunit: associates regular expressions with log type display fields; Storage mother unit: saves classification rules to the database cache module and preloads high-frequency regular expressions into memory; The log receiving and matching module is used for log receiving, matching, and classification, including: Log receiving subunit: Receives log data sent from the user host in real time and performs format standardization preprocessing; Real-time matching subunit: Based on the regular expression rules in the database, it matches the logs one by one and extracts the capture group fields; Cache subunit: preloads high-frequency regular expressions into memory, with matching latency ≤ 5ms; Storage subunit: stores the matched log category names and extracted fields into the database; Feedback sub-unit: Marks unmatched logs as "unknown type," triggers an alarm, and submits it to the AI ​​service to regenerate rules; Database module, used to store API configuration, classification rules and log data; Among them, each module communicates through HTTP protocol or SQL interface.

[0020] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A log audit unknown type log classification configuration method, characterized in that: The following steps are involved: Step 1: Enter the API token and base link of the AI ​​service through the AI ​​configuration page and store the data in the database; Step 2: Enter the original log sample into the log category management page to trigger the AI ​​interaction module to generate a named capture group regular expression; Step 3: Call the AI ​​service through HTTP request. The instruction is to convert the original log into a regular expression with named capture groups and parse the returned results. Step 4: Associate the generated regular expression with the custom log type and save it to the database; Step 5: The log receiving module matches the new log according to the classification information and automatically classifies it into the corresponding log type.

2. A log audit unknown type log classification configuration method according to claim 1, characterized in that: The AI ​​service in step 1 is a model based on natural language processing, and a regular expression is generated using the following formula: Regular expression = AI model (original log, instruction) The instruction includes a constraint condition of "removing the capital letter P after the question mark".

3. A log audit unknown type log classification configuration method according to claim 1, characterized in that: The named capture group regular expression generated in step 4 must pass the following validation: Syntax correctness check, using regular expression engine to verify the validity of syntax; Field mapping consistency check to ensure that the capture group names correspond to the log display fields. If the verification fails, an error message is returned and the AI ​​service is requested to generate again.

4. A log audit unknown type log classification configuration method according to claim 1, characterized in that: The association relationship in step 5 supports user-defined adjustment, including: Dynamically add and delete capture group fields; Set the matching threshold, the formula is: .

5. A log audit unknown type log classification configuration method according to claim 1, characterized in that: The AI ​​service is a natural language processing model based on the Transformer architecture, with a regular expression generation response time of ≤300ms and an accuracy rate of ≥98%.

6. A log audit unknown type log classification configuration method according to claim 1, characterized in that: The log receiving module in step 6 uses a real-time stream processing engine that supports Apache Flink or Kafka Streams, with a matching delay of ≤10ms.

7. A log audit unknown type log classification configuration method according to claim 1, characterized in that: The AI ​​service supports dynamic switching of multiple services. Configuration parameters include: AI service provider identification; Service priority weight, calculated as: W = 0.6 × response speed + 0.4 × accuracy.

8. A log audit unknown type log classification configuration system, using a log audit unknown type log classification configuration method according to any one of claims 1 to 7, characterized in that: Includes the following modules: AI configuration module: used to configure API tokens and basic links for AI services and store data in the database; Log classification management module: used to implement regular expression generation, field association and rule storage; Log receiving and matching module, used for log receiving, matching and classification; Database module, used to store API configuration, classification rules and log data; Among them, each module communicates through HTTP protocol or SQL interface.

9. A log audit unknown type log classification configuration system according to claim 8, characterized in that: The log classification management module includes: AI interaction subunit: calls external AI services to generate named capture group regular expressions; Regular expression association subunit: associates regular expressions with log type display fields; Storage mother unit: saves classification rules to the database. The cache module preloads high-frequency regular expressions into the memory.

10. A log audit unknown type log classification configuration system according to claim 8, characterized in that: The log receiving and matching module includes: Log receiving subunit: Receives log data sent from the user host in real time and performs format standardization preprocessing; Real-time matching subunit: Based on the regular expression rules in the database, it matches the logs one by one and extracts the capture group fields; Cache subunit: preloads high-frequency regular expressions into memory, with matching latency ≤ 5ms; Storage subunit: stores the matched log category names and extracted fields into the database; Feedback subunit: Marks unmatched logs as "unknown type," triggers an alarm, and submits it to the AI ​​service to regenerate rules.

Images