An offline log analysis system and method

Abstract

The application discloses an offline log analysis system and method. The system of the application comprises a log collection agent, a custom log processing service, an Elasticsearch cluster and a Kibana visualization platform; the log collection agent collects offline log files, the custom log processing service performs analysis and filtering processing on the offline log files; receives and stores a log analysis script uploaded by a user, and in response to an execution instruction of a stored log analysis script by the user, calls the script to analyze the processed log data. The Elasticsearch cluster stores the processed log data; the Kibana visualization platform queries and visually displays the stored log data. The application connects logs of various environments to the offline log platform of the company through a custom service platform as a channel, so that fast and efficient log analysis, statistics and query are realized.

Description

Technical Field

[0001] This invention relates to the field of network communication technology, and more specifically, to an offline log analysis system and method. Background Technology

[0002] In the field of internet communication, the conventional log platform architecture is ELK (Elasticsearch + Logstash + Kibana), but ELK (especially Logstash) has the following shortcomings: 1. The mismatch between real-time stream processing and offline batch processing: Logstash was originally designed for real-time data acquisition and pipeline processing. For existing, large amounts of offline log files (such as packaged .tar.gz files and historical log backups), Logstash requires complex configurations (such as using the sincedb mechanism of the file input plugin) to ensure non-duplicated acquisition and breakpoint resumption, which is cumbersome and inefficient in batch processing scenarios.

[0003] 2. Lack of Script Management: The ELK stack itself does not provide unified, persistent log analysis script management functionality. Users typically need to write painless scripts in Kibana or write external programs to query the Elasticsearch API. This approach cannot create reusable, one-click executable "analysis plans," resulting in the inability to consolidate and quickly reuse troubleshooting processes for complex issues.

[0004] 3. Limitations in data processing flexibility: Although Logstash boasts a rich plugin ecosystem, highly customized data processing needs (especially enterprise-specific encrypted field decryption, complex regular expression parsing, etc.) still require finding, installing, or even developing specific plugins. This process increases operational complexity and technical barriers.

[0005] 4. Resource Consumption and Deployment Complexity: As a JVM application, Logstash consumes a lot of memory and CPU when processing large amounts of historical data. In scenarios where only one-time or periodic import and analysis of offline logs is required, running a persistent Logstash service is not lightweight enough. Summary of the Invention

[0006] To address the shortcomings of the existing technology, the present invention aims to provide an offline log analysis system and method. The present invention uses a custom service platform as a channel to connect logs from various environments to the company's offline log platform, facilitating system operations including uploading log file packages and importing log analysis scripts, thereby enabling fast and efficient log analysis, statistical queries, and queries.

[0007] The technical solution of the present invention is described in detail below.

[0008] An offline log analysis system, comprising: A log collection agent, deployed at the log source, is used to collect offline log files and upload them to a custom log processing service. A custom log processing service communicates with the log collection agent, and includes offline log data parsing and analysis. The module and script management contingency plan module; the offline log data parsing and analysis module is used to receive the offline log files and parse and filter them; the script management contingency plan module receives and stores the log analysis scripts uploaded by the user; in response to the user's execution instruction for a stored log analysis script, the script is called to analyze the processed log data. An Elasticsearch cluster communicates with the custom log processing service to store log data processed by the custom log processing service. The Kibana visualization platform communicates with the Elasticsearch cluster and is used to query and visualize log data stored in the Elasticsearch cluster.

[0009] In this invention, the offline log data parsing and analysis module performs parsing and filtering processing on offline log files, including at least one of the following operations: 1) Match log content using regular expressions and perform format conversion on the matched content; 2) Decrypt the encrypted fields in the log; 3) The structured log text is parsed and split into multiple attribute fields; 4) Perform deletion, replacement, or renaming operations on specified fields.

[0010] In this invention, the script management pre-processing module stores the log analysis scripts uploaded by users as log analysis pre-processing plans and associates them with log scenarios or problem types.

[0011] In this invention, the log collection agent is configured to upload offline log files to the custom log processing service in batches according to a preset period or in response to a specific event.

[0012] The present invention also provides an offline log processing method, applied to the above-mentioned system, comprising: The log collection agent collects offline log files and uploads them. The custom log processing service receives the offline log file, parses and filters the offline log file; receives and stores the log analysis script uploaded by the user, and responds to the user's execution instruction for a stored log analysis script by calling the script to analyze the processed log data; Elasticsearch clusters store log data processed by a custom log processing service. Kibana is a visualization platform that allows users to query and visualize log data stored in an Elasticsearch cluster.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention replaces Logstash, which was designed for real-time streaming, with a specially developed "custom service." This service can more efficiently handle the uploading, decompression, parsing, and importing of batch offline files, avoiding the overhead and complexity of real-time streaming frameworks in batch processing scenarios.

[0014] 2. This invention introduces the concepts of an "offline log data parsing and analysis module" and a "script management contingency plan module," which script and template one-off, complex log analysis processes (such as multi-dimensional statistics and specific error pattern association). When similar problems recur, operations and maintenance personnel do not need to rewrite the analysis logic; they can simply "execute" the contingency plan script with one click, greatly improving the efficiency of problem investigation and the ability to accumulate knowledge.

[0015] 3. In this invention, "custom service" means that data parsing and filtering logic can be deeply customized according to the specific needs of actual business scenarios (such as integrating internal decryption SDKs and implementing specific data anonymization rules) without being limited by the functionality and compatibility of open source plugins.

[0016] 4. For offline analysis scenarios, the "custom service" in this invention can be designed to be more lightweight and focused, and may not need to run in the background. It can be started only when importing and analysis are needed, thereby saving system resources. Attached Figure Description

[0017] Figure 1 This is an architecture diagram of the offline log analysis platform of the present invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0019] This invention proposes a novel offline log analysis system, which includes a custom service, an Elasticsearch cluster, and a Kibana visualization platform.

[0020] Custom Service: The custom service is a Java backend service and is the core of this invention, containing two core functional modules: 1. Offline Log Data Parsing and Analysis Module: This module is the core of the data parsing and filtering logic. Users can predefine a series of data processing rules through configuration files or a web interface. These rules form a series of data processing pipelines, such as format conversion, field encryption / decryption, structured parsing, and field operations. Users can create and manage different offline log analysis and filtering rules according to their actual business needs.

[0021] For structured logs, Grok can be used in conjunction with regular expressions for filtering; unstructured logs (such as JSON or CSV formats) or special formats (multi-line, nested) can be analyzed and filtered using plugins such as JSON, CSV, and Multiline; specific filtering requirements and rules can be configured and managed in "Custom Services". For example: a) If the log contains timestamps or encrypted ciphertext (non-sensitive information), it can be matched and parsed into a visual time format and plaintext in the parsing rules according to this business scenario; b) Stack trace information in logs is often recorded in multiple lines, which is not convenient for querying and analysis. By configuring new parsing rules, multiple lines of stack trace records can be merged into one log line, which is convenient for subsequent log analysis and troubleshooting. c) If the logs contain a large amount of worthless field information, relevant rules can be configured to filter out this useless information.

[0022] 2. Script Management Plan Module: The script management plan module is key to achieving automated analysis and knowledge accumulation. Users can manage multiple frequently used script analysis plans. When a similar script log analysis is needed again, the plan can be executed with a single click, achieving rapid log analysis and troubleshooting. Example: In complex business scenarios that aggregate data from multiple sources, various intractable problems frequently arise requiring log analysis and troubleshooting. The log analysis process is complex, necessitating the analysis of logs from multiple modules within a specific time sequence. In such cases, the log analysis workflow can be scripted, imported into a custom service, and saved as a contingency plan. When troubleshooting is needed again, this script can be executed directly, achieving rapid and efficient problem analysis and resolution. The script management contingency plan module specifically implements the following functions: a) Script Upload and Storage: Provide a user interface (UI) or API to allow users to upload log analysis scripts. These scripts are stored in a database and associated with a name and description, forming a "log analysis plan"; b) Contingency Plan Management: Users can categorize and tag contingency plans (e.g., label them as "Performance Analysis", "Bug Tracking", "Security Audit"); c) Script Execution Engine: When a user selects a plan and triggers its execution through the interface, this engine calls the corresponding script. The script can query the Elasticsearch API to obtain pre-processed log data, perform complex statistical analysis, correlation analysis, or aggregation calculations, and return the final results to the user interface or generate a report.

[0023] Elasticsearch Cluster & Kibana Visualization Platform: These two components are inherited from a mature open-source ecosystem. Well-structured log data, processed by a custom service, is written to the Elasticsearch cluster. Users can then create index schemas in Kibana and explore, analyze, and visualize this offline log data through rich charts and dashboards, a process identical to using the standard ELK stack.

Claims

1. An offline log analysis system, characterized in that, include: A log collection agent, deployed at the log source, is used to collect offline log files and upload them to a custom log processing service. A custom log processing service, communicating with the log collection agent, includes an offline log data parsing and analysis module and a script management and contingency plan module. The offline log data parsing and analysis module receives the offline log files and parses and filters them. The script management and contingency plan module receives and stores log analysis scripts uploaded by users. In response to a user's execution command for a stored log analysis script, the module calls the script to analyze the processed log data. An Elasticsearch cluster communicates with the custom log processing service to store log data processed by the custom log processing service. The Kibana visualization platform communicates with the Elasticsearch cluster and is used to query and visualize log data stored in the Elasticsearch cluster.

2. The system according to claim 1, characterized in that, The offline log data parsing and analysis module performs at least one of the following operations on the parsing and filtering of offline log files: 1) Match log content using regular expressions and perform format conversion on the matched content; 2) Decrypt the encrypted fields in the log; 3) The structured log text is parsed and split into multiple attribute fields; 4) Perform deletion, replacement, or renaming operations on specified fields.

3. The system according to claim 1, characterized in that, The script management contingency plan module stores the log analysis scripts uploaded by users as log analysis contingency plans and associates them with log scenarios or problem types.

4. The system according to claim 1, characterized in that, The log collection agent is configured to upload offline log files to the custom log processing service in batches according to a preset period or in response to a specific event.

5. An offline log processing method, applied to the system as described in any one of claims 1-4, characterized in that, include: The log collection agent collects offline log files and uploads them. The custom log processing service receives the offline log file and parses and filters it. Receive and store log analysis scripts uploaded by users, and respond to user commands to execute a stored log analysis script by calling the script to analyze the processed log data; Elasticsearch clusters store log data processed by a custom log processing service. Kibana is a visualization platform that allows users to query and visualize log data stored in an Elasticsearch cluster.

Images