Automatic operation and maintenance event processing method and system based on log feature template

Abstract

The invention provides an operation and maintenance event automatic processing method and system based on a log feature template, and belongs to the field of computer system operation and maintenance, and the system comprises a template management module which is used for defining, editing and storing a feature template; the log acquisition module is used for acquiring logs from various sources; the preprocessing module is used for preliminarily analyzing and standardizing the collected logs; the message queue is used for transmitting data by adopting message middleware; the matching engine is used for carrying out real-time matching calculation on the preprocessed log and the feature template; the execution engine is used for executing the processing action defined in the feature template; the monitoring alarm module is used for monitoring the running state of the system; the storage module is used for storing a template database, an audit log and a script library; and the audit feedback module is used for persistently storing events and processing action execution records matched with the feature template to form an audit log. The system obviously improves the efficiency, accelerates the response speed, and avoids errors possibly caused by manual operation.

Description

Technical Field

[0001] This invention relates to the field of computer system operation and maintenance technology, specifically to an automatic processing method and system for operation and maintenance events based on log feature templates. Background Technology

[0002] In existing computer system operation and maintenance work, systems, applications, and services generate massive amounts of runtime logs. Operation and maintenance engineers need to continuously monitor these logs, promptly identify errors, warnings, or critical information, and take appropriate measures (such as restarting services, cleaning up space, and issuing alerts). Currently, the main problems are as follows:

[0003] 1. Manual monitoring is inefficient and prone to omissions: It relies on engineers to manually check log files or monitoring platforms. Faced with a large amount of log information, engineers are prone to fatigue and it is difficult to ensure 24 / 7 coverage without omissions, especially for occasional problems.

[0004] 2. Slow response time: There is a time delay from the discovery of a problem to manual intervention, which may cause the best time to deal with the problem to be missed and the problem to escalate.

[0005] 3. Repetitive work: For known common problems with fixed patterns (such as insufficient disk space, connection timeout for specific services, memory leak alarms, etc.), manual identification and handling are required each time, which consumes a lot of repetitive manpower.

[0006] 4. Existing tools lack flexibility: General log monitoring systems (such as the alarm function in the ELK Stack and Prometheus Alertmanager) are usually based on simple threshold or keyword matching. For scenarios that require combining multiple log line features or performing complex processing flows, the configuration is complex and inflexible, making it difficult to cover all known and automatically processed scenarios. Summary of the Invention

[0007] The purpose of this invention is to solve the problems existing in the prior art and provide an automatic processing method for operation and maintenance events based on log feature templates.

[0008] This invention is achieved through the following technical solution:

[0009] A first aspect of the present invention provides an automatic operation and maintenance event processing system based on log feature templates, the system comprising:

[0010] The template management module is used to define, edit, and store feature templates;

[0011] The log collection module is used to collect logs from various sources.

[0012] The preprocessing module is used to perform preliminary parsing and standardization on the collected logs;

[0013] Message queues are used to transmit data through message brokers;

[0014] The matching engine is used to perform real-time matching calculations between preprocessed logs and feature templates;

[0015] The execution engine is used to execute the processing actions defined in the feature template;

[0016] The monitoring and alarm module is used to monitor the system's own operating status;

[0017] The storage module is used to store the template database, audit logs, and script library;

[0018] The audit feedback module is used to persistently store the records of events and processing actions matched by feature templates, forming an audit log.

[0019] Furthermore, the template management module is also used to provide graphical and configuration file interfaces, query feature templates, and archive feature templates.

[0020] Furthermore, the required fields for the standard output data format after standardization include: timestamp, hostname, application name, log level, and log message.

[0021] Another aspect of the present invention provides an automatic processing method for operation and maintenance events based on log feature templates, wherein the system is used to identify and process operation and maintenance events, and the method includes:

[0022] Step 1: Define a feature template for each known and automatable operation and maintenance event;

[0023] Step 2: Collect log data and preprocess the logs based on log parsing libraries and regular expressions;

[0024] Step 3: Based on the matching logic defined by the feature template, match the feature matching items in the key features of the log data with the feature template;

[0025] Step 4: Execute the processing action of the feature template configuration that successfully matches the log data.

[0026] Furthermore, the feature template in step 1 includes: a unique template ID that identifies the template, a description of the event that the template targets, key processing features of the template, processing parameters, template effective time, and suppression strategy.

[0027] Furthermore, the key features of the log data include one or more feature matching items, which include: log source identifier, log level, expression used to match the log line content, the position of the expression in the log line, the number of times the feature matching item needs to be successfully matched, the logical relationship between multiple feature matching items, and the processing action automatically executed by the system.

[0028] Furthermore, the template effective time and suppression strategy include defining the time period during which the template is effective and the time period during which matching will not be repeated after a successful match.

[0029] Furthermore, step 3 includes: applying hierarchical filtering to the matching engine, i.e., first filtering feature templates that do not meet the requirements according to log source and level; then creating a keyword index for the remaining feature templates; and using an inverted index for matching.

[0030] Furthermore, the operations of the inverted index include:

[0031] S1. Initialize an empty inverted index dictionary;

[0032] S2. Classify each feature match for each template, and apply different strategies based on the type:

[0033] If the feature match is an exact string, then the exact string is used as the key, and the template ID is added to the corresponding value set;

[0034] If the feature match is a regular expression and has a fixed string, then the fixed string is used as the key and the template ID is added to the corresponding value set; if the feature match is a regular expression and does not have a fixed string, then the template is added to a special list.

[0035] Furthermore, step 4 includes the following operations:

[0036] Execute a predefined script;

[0037] Send notifications, including notifications of processing completion results and notifications of processing progress;

[0038] Record audit logs, which include: start time, action performed, parameters, execution result, and output information;

[0039] Implement processing strategies based on the execution results, including: failure processing strategy: trigger a higher-level alarm; success processing strategy: mark the process as automatically processed in the audit log.

[0040] The beneficial effects of this invention are: significantly improved efficiency: automating the processing of a large number of repetitive, known-pattern operation and maintenance events, freeing up engineers' energy to focus on more complex problems; faster response speed; avoiding errors that may be caused by manual operation; high flexibility: the template-based configuration method allows engineers to freely define various complex matching rules and processing flows according to actual operation and maintenance scenarios, with a wide coverage; complete audit logs record all automated processing processes, facilitating backtracking and analysis; low implementation cost: mainly based on rule matching and script execution, the core engine can be built using existing mature open-source components (such as Logstash / Fluentd for collection, Redis for caching templates, and task queues for execution), making development difficulty and resource consumption relatively controllable. Attached Figure Description

[0041] Figure 1 This is a diagram of the overall system architecture of the present invention.

[0042] Figure 2 This is an example diagram defining the feature template of the present invention.

[0043] Figure 3 This is an example diagram illustrating how the inverted index accelerates matching in this invention. Detailed Implementation

[0044] The present invention will now be described in further detail with reference to the accompanying drawings:

[0045] This invention provides an automated operation and maintenance event processing system based on log feature templates. The system architecture is as follows: Figure 1 As shown, it includes:

[0046] 1) Template Management Module: Provides a graphical or configuration file interface for engineers to define, edit, enable / disable feature templates in the operation interface. Historical versions of feature templates can be queried and archived.

[0047] 2) Log collection module; collects logs from various sources, including: Filebeat collector; Syslog client; Kafka consumer.

[0048] 3) Preprocessing module: Performs preliminary parsing and standardization on the collected logs. Required fields in the standard output data format after preprocessing include: timestamp, hostname, app_name, log_level, and message.

[0049] (Log message).

[0050] 4) Message Queues (Message Middleware): Kafka and RabbitMQ are used as message middleware to transmit data.

[0051] 5) Matching Engine: The log collector pushes data to the matching engine through a message queue (such as Kafka). The matching engine loads feature templates and performs real-time matching calculations on the preprocessed logs.

[0052] 6) Execution Engine: Executes the processing actions defined in the template (script execution, notification sending, failure retry, etc.) securely and reliably.

[0053] 7) Monitoring and Alarm Module: By recording the matching event logs and action execution logs of the feature templates, the module monitors the system's own operating status (such as whether the matching engine is alive and whether the action execution has timed out / failed) and issues alarms.

[0054] 8) Storage module: Stores template database, audit logs, script library. Other modules access the storage module through a unified database API.

[0055] 9) Audit Feedback Module: Persistently store all event and processing action execution records that match the template to form audit logs for statistical analysis.

[0056] This invention also provides an automatic processing method for operation and maintenance events based on log feature templates, comprising the following steps:

[0057] 1. Feature Template Definition

[0058] Based on historical experience, engineers manually analyze the corresponding log output patterns for known and automatable operational events, including insufficient disk space, abnormal service process exits, database connection pool exhaustion, and frequent timeouts of specific API interfaces. A characteristic template is defined for each event, and these templates are stored in a template database within the storage module. For example... Figure 2 As shown in the example, the rules of the feature template include:

[0059] (1) Unique template ID: Identifies the template.

[0060] (2) Event description: Clearly describe the event that this template is for.

[0061] (3) Key Features: Contains one or more feature matches. Each feature match consists of the following:

[0062] 1) Log source identifier: Specifies the log source (such as hostname, application name, service name).

[0063] 2) Log levels: such as ERROR, WARN, INFO.

[0064] 3) Expressions: Used to match the core features of log line content. Expressions can be exact keywords, wildcards, or regular expressions.

[0065] 4) Matching Position: Specifies the position of the keyword / regular expression in the log line, including the beginning, middle, and end of the line. The regular expression supports common patterns, such as .* matching any character, for example:

[0066] \d+ matches numbers, \w+ matches word characters, etc.

[0067] 5) Match count: Specifies the number of times the feature needs to be matched successfully, including: single occurrence, consecutive occurrence N times, and occurrence N times within M seconds.

[0068] 6) Matching logic: Define the logical relationship between multiple feature matching items, including AND, OR, NOT; furthermore, it can also support compound logic, such as AND-OR combination.

[0069] 7) Processing Actions: Define one or more actions that the system will automatically execute when a template match is successful. These actions include, but are not limited to:

[0070] a) Execute predefined Shell / Python scripts;

[0071] b) Send a notification, the content of which is defined in the feature template of the event match.

[0072] c) Record the processing events to the audit log.

[0073] d) Handling strategies, including failure handling strategy: trigger a higher-level alarm; success handling strategy: mark the event as "automatically handled", and after marking "automatically handled" in the audit log, the event will not be matched again.

[0074] (4) Processing parameters: Pass the required parameters to the processing action, such as: the directory path to be cleaned, the name of the service to be restarted, and the list of notification recipients.

[0075] (5) Effective Time / Suppression Strategy: Define the time period during which the feature template is effective or the period after a successful match (e.g., 300 seconds) during which the match will not be repeated (to prevent loop triggering). The format of the time period is defined as a time expression and a time interval, such as 9:00-18:00 on weekdays.

[0076] 2. Log Collection and Analysis

[0077] The log collection module collects log data from configured log sources (files, Syslog, Kafka, etc.) in real-time or near real-time (with a collection latency of 1-5 seconds). Specifically, the log source is defined in the system configuration file and supports file paths, network interfaces, message queues, etc.; it is obtained through automatic collection interfaces, such as Filebeat for files, Syslog clients for system logs, and Kafka consumers for message queues; preferably, it can also be manually uploaded via a page.

[0078] The preprocessing module preprocesses the logs, including: timestamp parsing (converting to a standard format), log level extraction, source identifier (hostname, application name) extraction, message content structuring (e.g., JSON parsing), and data standardization. This preprocessing is based on existing log parsing libraries such as Grok and regular expressions, which will not be elaborated upon here. Abnormal logs detected during preprocessing will be marked as parsing failures, stored in an exception log database, and trigger an alert to notify engineers.

[0079] 3. Match logs with feature templates.

[0080] After collecting logs, the log collection module triggers the matching engine in real time to match the preprocessed log data with all preloaded feature templates. Specifically, based on the matching logic defined in the feature templates, it checks and matches feature matching items in the key features of the logs. Specifically, the matching engine uses hierarchical filtering: first, it quickly filters templates by log source and level; then, it builds a keyword index for the remaining templates and uses an inverted index to accelerate matching, avoiding traversing each log entry one by one.

[0081] The inverted index building algorithm is as follows: Figure 3 As shown, the steps are as follows:

[0082] S1. Initialize an empty inverted index dictionary.

[0083] Figure 3 The corresponding Chinese code is: the __init__ method:

[0084] When an InvertedIndex object is created, its constructor initializes two empty data structures:

[0085] `self.index` is an empty inverted index dictionary. Its keys will be keywords (strings), and its values ​​will be a set containing all template IDs that have that keyword.

[0086] `self.no_index_templates` is the "list of templates that cannot be indexed". In the code, it is implemented using a set to record the template IDs from which no index keywords can be extracted for any of the features.

[0087] S2. Classify each feature match for each feature template using different strategies:

[0088] If the keyword of the feature matching item is an exact string, then the exact string is used as the key, and the template ID is added to the corresponding value set.

[0089] If the keyword of the feature is a regular expression, then attempt to extract a fixed string (at least 3 characters) from the regular expression. For each extracted fixed string, use that string as the key and add the template ID to the corresponding value set. If there is no fixed string in the regular expression, then add the template to a special list, namely the list of templates that cannot be indexed.

[0090] like Figure 3 As shown, this core step is implemented by the build_index method:

[0091] The code uses the feature.is_exact attribute to determine the feature type:

[0092] If it is an exact string, the add_keyword method is called directly, passing in the string (feature.keyword) and the ID of the current template (template.id); at the same time, the has_index flag is set to True, indicating that this template has been successfully indexed.

[0093] For regular expression features, the code calls the `extract_fixed_strings` method for processing: using the regular expression `r'[a-zA-Z0-9_]{3,}'`, it extracts all consecutive sequences of letters, numbers, or underscores with a length of at least 3 from a complex regular expression pattern. If a fixed string is extracted (if `fixed_strings` is true), each extracted string `s` is used as a keyword and added to the index along with the template ID using a loop; similarly, if any string is successfully extracted and added, `has_index` is marked as True.

[0094] Furthermore, if the regular expression does not contain a fixed string, the template is added to a special list, such as... Figure 3 After the inner loop (iterating through all features) finishes, check the has_index flag. If the flag is still False, it means that after iterating through all features of the template, no indexable keyword was successfully extracted for any of them (i.e., no exact string was extracted, and all regular expression features failed to extract a fixed string); in this case, add the ID of the template to the self.no_index_templates collection.

[0095] When a log entry fully meets all the matching conditions of a template, it is determined that it is a successful match with that template.

[0096] Furthermore, if a log time matches multiple templates simultaneously, the matching will be performed in the order of the pre-set template priority.

[0097] Furthermore, if the matching logic contains the NOT operator, it means excluding specific keywords. For example, NOT keyword "test" means matching log lines that do not contain "test".

[0098] Upon successful matching, an event signal will be generated, the log data status will be recorded, and the execution process will be triggered.

[0099] 4. Perform processing actions

[0100] If the log matches the feature template successfully, an event signal containing the template ID and the matching result will immediately trigger the action execution engine to execute the processing action configured for that template. This action may include, but is not limited to:

[0101] a) Execute predefined Shell / Python scripts, specifically calling scripts from the system's script library from the paths defined in the event feature template, such as: cleaning up temporary files in a specified directory, restarting a specific service, or expanding the disk LVM.

[0102] b) Send Notification: Select whether to send the notification via email or SMS, or both. The notification includes notifications of completion results and notifications of the processing process; the content of the notification is defined in the event feature template.

[0103] c) Record the processing event to the audit log, which includes: start time, action performed, parameters, execution result - success / failure, and output information.

[0104] d) Handling strategies, including failure handling strategy: trigger a higher-level alarm; success handling strategy: mark the event as "automatically handled", and after marking "automatically handled" in the audit log, the event will not be matched again.

[0105] The failure handling strategy is as follows: upon execution failure, the alarm is escalated according to the failure handling strategy configured in the template. The alarm escalation logic employs a retry mechanism: after the initial alarm, the processing action is restarted, and it is determined whether the retry execution was successful. If the number of execution failures exceeds a threshold (e.g., 3 consecutive execution failures), the alarm is escalated; the alarm is sent via email, SMS, or an external integrated alarm platform. Upon successful execution, a success notification is sent, and the event status is updated.

[0106] 5. Audit and Feedback

[0107] All events and processing actions that match the template are persistently stored. Specifically, log events are automatically generated when they successfully match the feature template and when actions are executed. The logs are stored in JSON format and include fields such as event ID, timestamp, action, and result. They are stored in the system's audit database to form audit logs, which engineers can view in the user interface to view the automatic processing history, success rate, and failure reason statistics and analysis.

[0108] Preferably, engineers can iteratively optimize the feature template definition (adjust matching rules and processing actions) based on the actual operating results, and automatically match template versions and record modification history.

[0109] Example: The present invention will be described in detail using the example of automatically handling the disk space shortage alarm of the / var / log partition.

[0110] 1. Define feature template:

[0111] ID:DISK_FULL_VAR_LOG

[0112] Event Description: Automatically handles low disk space warning for / var / log partition.

[0113] Key features:

[0114] -Feature 1:

[0115] Log source: host-* # Matches all hosts

[0116] Log level: ERROR

[0117] Keywords: "No space left on device" AND " / var / log"

[0118] Matching position: Contains

[0119] Number of matches: 1

[0120] -Feature 2:

[0121] Log source: host-*

[0122] Log level: INFO# or WARN, depending on the specific log file.

[0123] Keywords: "Filesystem.*%.* / var / log" # Regular expression matches the position of the line in the df output: containing

[0124] Number of matches: 1

[0125] Matching logic: Feature 1 AND Feature 2 # Both features must be satisfied simultaneously

[0126] Processing actions:

[0127] - Action type: Execute script

[0128] Script path: clean_var_log.sh

[0129] Parameters: ["--age-days","7","--extensions","*.log,*.gz"]# Clean up .log and .gz files older than 7 days.

[0130] - Action type: Send notification

[0131] Notification message: "The / var / log partition on host {hostname} is running out of space. Log files older than 7 days have been automatically cleaned up. Please monitor for future space changes."

[0132] 2. Log Collection and Analysis

[0133] The log collection module collects system logs from the server, specifically through a Filebeat proxy.

[0134] System logs and application logs are collected in real time from files such as / var / log / syslog.

[0135] The preprocessing module parses the logs, extracting fields such as hostname, log level, and message content. Specifically, it uses Grok mode to parse log lines, extracting fields such as timestamp, hostname, log_level, and message, standardizing them into JSON format, and outputting them to a Kafka topic.

[0136] 3. Match template

[0137] The matching engine loads all enabled templates. When a log entry has an ERROR level and contains "No space left on device" and " / var / log", and is associated with another log entry (possibly the output of the `df` command) containing information like "Filesystem...%... / var / log" (indicating high partition usage), the matching engine determines that the `DISK_FULL_VAR_LOG` template has matched successfully. This other log entry is automatically associated by the system with logs from the same host group within the same time period.

[0138] In the specific process, the matching engine obtains log data from Kafka, first filters logs with log levels of ERROR and INFO, and then uses keyword indexing to match feature 1 and feature 2. When the same host matches both features within 5 seconds, it is determined that the template match is successful, an event signal is generated, and the execution engine is triggered.

[0139] 4. Perform processing actions

[0140] After a successful match, the action execution engine retrieves the script path and parameters from the storage module based on the template ID and calls the clean_var_log.sh script. It then calls the script execution service via an internal RPC interface, passing in the cleanup file, which includes the number of days to clean up and the file extension. The action execution engine passes the parameters to the script command line, and the script parses the parameters to perform the cleanup. Simultaneously, the action execution engine sends a notification to the specified email address.

[0141] The above matching template, script process, script execution, output (standard output and error output of script execution), result (success), and notification sending records are all written to the database.

[0142] The above technical solution is only one embodiment of the present invention. For those skilled in the art, based on the principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the technical solutions described in the specific embodiments of the present invention. Therefore, the foregoing description is only a preferred option and is not restrictive.

Claims

1. An automated operation and maintenance event processing system based on log feature templates, characterized in that: The system includes: The template management module is used to define, edit, and store feature templates; The log collection module is used to collect logs from various sources. The preprocessing module is used to perform preliminary parsing and standardization on the collected logs; Message queues are used to transmit data through message brokers; The matching engine is used to perform real-time matching calculations between preprocessed logs and feature templates; The execution engine is used to execute the processing actions defined in the feature template; The monitoring and alarm module is used to monitor the system's own operating status; The storage module is used to store the template database, audit logs, and script library; The audit feedback module is used to persistently store the records of events and processing actions matched by feature templates, forming an audit log.

2. The automatic operation and maintenance event processing system based on log feature templates according to claim 1, characterized in that: The template management module also provides graphical and configuration file interfaces for querying feature templates. Archive the feature templates.

3. The automatic operation and maintenance event processing system based on log feature templates according to claim 1, characterized in that: The required fields for the standard output data format after standardization include: timestamp, hostname, application name, log level, and log message.

4. An automatic processing method for operation and maintenance events based on log feature templates, using the system described in any one of claims 1-3 to identify and process operation and maintenance events, characterized in that: The method includes: Step 1: Define a feature template for each known and automatable operation and maintenance event; Step 2: Collect log data and preprocess the logs based on log parsing libraries and regular expressions; Step 3: Based on the matching logic defined by the feature template, match the feature matching items in the key features of the log data with the feature template; Step 4: Execute the processing action of the feature template configuration that successfully matches the log data.

5. The automatic processing method for operation and maintenance events based on log feature templates according to claim 4, characterized in that: The feature template in step 1 includes: a unique template ID that identifies the template, a description of the event that the template targets, key processing features of the template, processing parameters, template effective time, and suppression strategy.

6. The automatic processing method for operation and maintenance events based on log feature templates according to claim 5, characterized in that: The key features of the log data include one or more feature matching items, which include: log source identifier, log level, expression used to match the log line content, the position of the expression in the log line, the number of times the feature matching item needs to be matched successfully, the logical relationship between multiple feature matching items, and the processing action automatically executed by the system.

7. The automatic processing method for operation and maintenance events based on log feature templates according to claim 5, characterized in that: The template effective time and suppression strategy include defining the time period during which the template is effective and the time period during which matching will not be repeated after a successful match.

8. The automatic processing method for operation and maintenance events based on log feature templates according to claim 4, characterized in that: Step 3 includes: applying hierarchical filtering to the matching engine, i.e. filtering feature templates that do not meet the requirements according to log source and level; then creating a keyword index for the remaining feature templates; and using an inverted index for matching.

9. The automatic processing method for operation and maintenance events based on log feature templates according to claim 8, characterized in that: The operations of the inverted index include: S1. Initialize an empty inverted index dictionary; S2. Classify each feature match for each template, and apply different strategies based on the type: If the feature match is an exact string, then the exact string is used as the key, and the template ID is added to the corresponding value set; If the feature match is a regular expression and has a fixed string, then the fixed string is used as the key and the template ID is added to the corresponding value set; if the feature match is a regular expression and does not have a fixed string, then the template is added to a special list.

10. The automatic processing method for operation and maintenance events based on log feature templates according to claim 5, characterized in that: The operations in step 4 include: Execute a predefined script; Send notifications, including notifications of processing completion results and notifications of processing progress; Record audit logs, which include: start time, action performed, parameters, execution result, and output information; Implement processing strategies based on the execution results, including: failure processing strategy: trigger a higher-level alarm; success processing strategy: mark the process as automatically processed in the audit log.

Images