Intelligent log analysis system and method based on three-program separation architecture
By employing a three-program separation architecture and artificial intelligence technology, the performance bottlenecks and resource waste issues of traditional log collection systems have been resolved, enabling efficient log parsing and dynamic optimization, thereby improving system performance and resource utilization.
Patent Information
- Application Number
- CN202511539185.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-27
- Publication Date
- 2026-02-27
AI Technical Summary
Traditional log collection systems suffer from performance bottlenecks due to single-unit parsers, low rule matching efficiency, lack of adaptability in static configuration, difficulty in updating parsing rules, and lack of intelligent optimization mechanisms, resulting in performance degradation and resource waste.
It adopts a three-program separation architecture, including a forwarding program, a parsing program, and a monitoring program. It utilizes feature matching and artificial intelligence technologies to achieve intelligent log distribution, dynamic parsing rule optimization, and performance improvement, and supports hot deployment and dynamic load balancing.
It significantly improves log parsing performance, reduces the number of rule checks, increases resource utilization and system availability, achieves efficient log parsing and dynamic optimization, and supports flexible deployment modes.
Smart Images
Figure CN121579299A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of log processing, more particularly to an intelligent log analysis system and method based on a three-program separation architecture. BACKGROUND
[0002] A log collection and analysis system provides core capabilities such as security monitoring, fault troubleshooting, and compliance auditing for enterprises by comprehensively collecting, processing, and analyzing log data, and is a key tool for digital operation and security assurance. The functions of the log collection and analysis system include log collection (multi-source adaptation), storage (structured / unstructured), and analysis (format standardization), providing a basis for real-time monitoring, abnormal alarm, report visualization, and compliance check; its significance lies in providing security assurance, operation optimization, compliance auditing, and providing business insight, specifically, discovering potential threats (such as abnormal login and data leakage) in real time, shortening attack response time; locating system fault causes (such as performance bottlenecks and service crashes) through logs, improving stability; meeting industry regulation requirements, automatically generating log retention and audit reports; analyzing user behavior logs (such as access paths and operation habits), assisting product optimization and decision-making. Therefore, the log collection and analysis system is the "nerve center" of enterprise digital transformation, its functions cover the whole life cycle management of data, and its core value lies in transforming massive logs into actionable security and business insights.
[0003] However, the traditional log collection system has the following technical problems: Single parser performance bottleneck: the traditional log parser uses a single architecture, and when multiple Grok parsing rules are configured, each log needs to be matched with all rules one by one, and in the worst case, a log needs to be checked by 100 rules before it is determined that it cannot be parsed, resulting in a great performance loss.
[0004] Low efficiency of rule matching: the existing system cannot pre-identify which logs are suitable for which parsing rules, resulting in a large number of invalid matching attempts and a serious waste of CPU resources.
[0005] Static configuration lacks adaptability: the traditional system uses static configuration, which cannot dynamically adjust the parsing strategy according to the characteristics of the logs, and invalid parsing attempts will still be performed for logs with obvious characteristics but no parsing value.
[0006] Difficult to update parsing rules: the update of parsing rules in the existing system requires restarting the service, which cannot realize hot deployment, affecting the continuity and availability of the system.
[0007] Lack of intelligent optimization mechanism: the existing system cannot automatically identify and optimize inefficient parsing rules, nor can it select the optimal parsing method according to the structure characteristics of the logs.
[0008] Therefore, how to realize intelligent log distribution, dynamic analysis rule optimization and performance improvement based on program separation-based log analysis architecture is an urgent problem to be solved. SUMMARY
[0009] The application aims to provide a three-program separation architecture-based intelligent log analysis system and method, which can realize intelligent log distribution, dynamic analysis rule optimization and performance improvement based on feature recognition.
[0010] The application provides a three-program separation architecture-based intelligent log analysis system, which comprises a forwarding program module and an analysis program module.
[0011] The application also provides a three-program separation architecture-based intelligent log analysis method, which uses the above-mentioned three-program separation architecture-based intelligent log analysis system to analyze logs.
[0012] The three-program separation architecture-based intelligent log analysis system and method provided by the application have the following advantages: The three-program separation architecture is proposed for the first time, which decomposes a monolithic log parser into a forwarding program, an analysis program and a monitoring program, fundamentally solving the performance bottleneck problem of traditional solutions. The use of artificial intelligence technology to automatically optimize the analysis program can automatically select the optimal analysis strategy according to the log features. The system can automatically optimize the analysis strategy according to the log features, automatically select the optimal analysis method for structured logs such as JSON, and improve the analysis efficiency by more than 300%, showing superior dynamic optimization capability. The use of an interpreted language to implement the analysis program supports code modification without the need for compilation and restart to take effect, and the system availability reaches more than 99.9%, with good hot deployment capability. Based on log features, the feature database is used to realize accurate log distribution, avoid the performance loss of multiple rule checks for a log in traditional solutions, avoid invalid processing, improve resource utilization by more than 50%, and realize feature-driven intelligent distribution. The three-program separation architecture supports independent deployment and expansion, can be flexibly configured according to different environment requirements (networked / offline deployment), supports two deployment modes of networking and offline, meets the deployment requirements of different security levels, realizes a hybrid deployment architecture, and has strong expansibility. The intelligent log analysis system based on the three-program separation architecture can realize intelligent log distribution, dynamic analysis rule optimization and performance improvement based on feature recognition, thereby fundamentally solving the performance bottleneck problem of a traditional single log parser, and has important technical value and broad application prospect. BRIEF DESCRIPTION OF DRAWINGS
[0013] The application will be further described below in combination with the drawings and embodiments, and the drawings are as follows: Figure 1 is the composition block diagram of the intelligent log analysis system based on the three-program separation architecture provided by the application; Figure 2 is the flowchart of the intelligent log analysis method based on the three-program separation architecture provided by the application; Figure 3 is the core algorithm flow of the forwarding program provided by the application; Figure 4 is the deployment architecture diagram of the networking deployment mode provided by the application; Figure 5 is the deployment architecture diagram of the offline deployment mode provided by the application. DETAILED DESCRIPTION
[0014] In order to have a clearer understanding of the technical features, purposes and effects of the application, the specific embodiments of the application will be described in detail below with reference to the drawings.
[0015] Figure 1 The composition block diagram of the intelligent log analysis system based on the three-program separation architecture of the embodiment is shown. In the embodiment, the intelligent log analysis system based on the three-program separation architecture comprises a forwarding program module and an analysis program module; the forwarding program module is used for receiving original logs and distributing them to corresponding analysis programs according to feature matching; and the analysis program module is used for analyzing the received logs to obtain formatted log data and sending the formatted log data to a target platform. In an exemplary embodiment, the forwarding program module is configured to: According to the logs sent by the log source and the feature database, a feature matching engine is used for feature matching to obtain a matching result; According to the matching result, an intelligent filter is used to filter out logs that cannot be analyzed to obtain to-be-analyzed logs; According to the to-be-analyzed logs, a kafka pipeline management method is used to create an independent message queue for each analysis program; In an exemplary embodiment, the feature database stores log feature patterns corresponding to different parsing programs; The feature matching engine supports prefix matching, suffix matching, and regular expression matching feature recognition methods. In an exemplary embodiment, the parsing program module is configured to: receive logs in the message queue, parse the logs using multiple parsing methods to obtain parsing results; format the parsing results to obtain formatted log data; send the formatted log data to a target platform; In an exemplary embodiment, the multiple parsing methods include Grok parsing, JSON parsing, and regular expression parsing. In an exemplary embodiment, the intelligent log parsing system based on the three-program separation architecture further includes a monitoring program module for queue backlog detection and log analysis, and for optimizing parsing code and updating the feature database based on the detection and analysis results. In an exemplary embodiment, the monitoring program module includes a queue monitoring submodule, an AI analysis engine, a code optimizer, and a feature library manager. In an exemplary embodiment, the queue monitoring submodule is used for queue backlog detection of the message queue, to confirm that the message queue has backlog, to sample the parsing program code and sample the backlog logs. In an exemplary embodiment, the AI analysis engine is used for analyzing the log samples to identify parsing efficiency problems. In an exemplary embodiment, the code optimizer is used to generate optimized code based on the parsing efficiency problems, and to deploy new parsing programs based on the optimized code. In an exemplary embodiment, the feature library manager is used to update the feature database based on the parsing efficiency problems.
[0016] The present embodiment provides an intelligent log parsing method based on a three-program separation architecture, which utilizes the intelligent log parsing system based on the three-program separation architecture described above to parse logs.
[0017] In some embodiments, the intelligent log parsing method based on the three-program separation architecture described above can also be implemented in the following manner.
[0018] As Figure 2 The flowchart of the intelligent log parsing method based on the three-program separation architecture is shown; in this embodiment, the traditional monolithic log parser is divided into three independently running program modules, namely a forwarding program, a parsing program, and a monitoring program, and artificial intelligence technology is used to achieve intelligent log processing and system optimization.
[0019] 1. Three-program separation architecture design 1.1 Forwarding program The forwarding program serves as the core scheduling module of the system, responsible for receiving raw logs and distributing them to corresponding parsing programs based on feature matching: • Implements feature library management, storing log feature patterns corresponding to different parsing programs; • Supports prefix matching, suffix matching, regular expression matching, and other feature recognition methods; • Integrates Kafka pipeline management to create independent message queues for each parsing program; • Implements an intelligent filtering mechanism to avoid sending logs that cannot be parsed to parsing programs; The core algorithm flow of the forwarding program is shown in Figure 3 .
[0020] 1.2 Parsing program The parsing program is implemented in an interpreted language, supporting hot deployment and dynamic modification: • Based on fixed templates, containing a four-step processing flow; • Written in an interpreted language, modifications take effect without compilation; • Supports multiple parsing methods: Grok parsing, JSON parsing, regular expression parsing; • Implements independent Kafka consumers to handle specific types of logs.
[0021] The standard template for the parsing program is shown in Table 1: Table 1: Standard template for parsing program
[0022] 1.3 Monitoring program The monitoring program uses artificial intelligence technology to be responsible for the intelligent optimization and dynamic adjustment of the system: • Real-time monitoring of the queue backlog of Kafka pipelines corresponding to each parsing program; • Analyzing log samples based on AI technology to identify parsing efficiency issues; • Automatically optimizing the code logic and parsing mode of the parsing program; • Providing feature recognition rules for the forwarding program to achieve intelligent filtering.
[0023] 2. Intelligent feature recognition and filtering mechanism 2.1 Feature Database Design The system maintains a feature database to store the following information: • Parser ID and corresponding log feature pattern; • Unresolvable log features (negative features); • Statistical information and performance indicators of feature matching.
[0024] 2.2 Feature Matching Algorithm The forwarding program adopts a multi-level feature matching mechanism as shown in Table 2: Table 2: Forwarding Program
[0025] 3. AI-driven Intelligent Optimization Mechanism 3.1 Queue Backlog Detection and Optimization The monitoring program continuously monitors the queue depth of each Kafka pipeline, as shown in Table 3: Table 3: Queue Backlog Detection and Optimization Program
[0026] 3.2 Parser Intelligent Optimization When the monitoring program discovers that the parser processes structured logs (such as JSON), it automatically optimizes the parsing logic, as shown in Table 4: Table 4: Parser Intelligent Optimization Program
[0027] 4. Dynamic Load Balancing Mechanism 4.1 Pipeline State Management The system implements a dynamic pipeline management mechanism: • When a Kafka pipeline has no log input for a long time, the corresponding parser is automatically disabled; • When the disabled pipeline has log input again, the corresponding parser is automatically started; • Dynamically adjust the number of instances of the parser according to the amount of log input; 4.2 Load Balancing Advantages Compared with traditional solutions, this architecture has the following advantages: • Avoids the performance loss of "one log being checked by multiple rules"; • Achieves efficient distribution of "log directly to corresponding parser"; • Supports independent scaling and optimization of parsers; Figure 4 is the deployment architecture diagram of the networking deployment mode; Figure 5 is the deployment architecture diagram of the offline deployment mode.
[0028] In some embodiments, the intelligent log parsing method based on the three-program separation architecture described above can also be implemented in the following way.
[0029] Embodiment 1: Feature matching implementation of the forwarding program Step 1: Feature library initialization, as shown in Table 5: Table 5: Feature library initialization program
[0030] Step 2: Log distribution logic, as shown in Table 6; Table 6: Log distribution logic
[0031] Embodiment 2: Interpretive language implementation of the parsing program Parsing program configuration template, as shown in Table 7: Table 7: Parsing program configuration template
[0032] Dynamic parsing program generation, as shown in Table 8: Table 8: Dynamic parsing program generation
[0033] Embodiment 3: AI optimization implementation of the monitoring program Queue monitoring and analysis, as shown in Table 9: Table 9: Queue monitoring and analysis
[0034] AI-driven parsing optimization, as shown in Table 10: Table 10: Parsing optimization
[0035] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the specific embodiments described above, which are only illustrative and not limiting. Those of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, which are all within the protection of the present application.
Claims
1. An intelligent log parsing system based on a three-program separation architecture, characterized in that, It includes a forwarding module and a parsing module; the forwarding module is used to receive raw logs and distribute them to the corresponding parsing program according to feature matching; the parsing module is used to parse the received logs to obtain formatted log data, and send the formatted log data to the target platform.
2. The intelligent log parsing system based on a three-program separation architecture according to claim 1, characterized in that, The forwarding program module is configured as follows: Based on the logs sent by the log source and the feature database, feature matching is performed using a feature matching engine to obtain the matching results; Based on the matching results, use a smart filter to filter out logs that cannot be parsed, and obtain the logs to be parsed; Based on the logs to be parsed, an independent message queue is created for each parser using Kafka pipeline management methods.
3. The intelligent log parsing system based on a three-program separation architecture according to claim 2, characterized in that, The feature database stores log feature patterns corresponding to different parsing programs; the feature matching engine supports feature recognition methods such as prefix matching, suffix matching, and regular expression matching.
4. The intelligent log parsing system based on a three-program separation architecture according to claim 1, characterized in that, The parsing program module is configured as follows: Receive logs from the message queue, parse the logs using multiple parsing methods, and obtain the parsing results; The parsing results are formatted to obtain formatted log data; The formatted log data is sent to the target platform.
5. The intelligent log parsing system based on a three-program separation architecture according to claim 4, characterized in that, The various parsing methods include Grok parsing, JSON parsing, and regular expression parsing.
6. The intelligent log parsing system based on a three-program separation architecture according to claim 1, characterized in that, The intelligent log parsing system based on a three-program separation architecture also includes a monitoring program module, which is used to perform queue backlog detection and log analysis, and optimize the parsing code and update the feature database based on the detection and analysis results.
7. The intelligent log parsing system based on a three-program separation architecture according to claim 6, characterized in that, The monitoring program module includes a queue monitoring submodule, an AI analysis engine, a code optimizer, and a feature library manager. The queue monitoring submodule is used to detect backlog in the message queue, confirm that there is a backlog, sample the parsing program code, and sample the backlog logs. The AI analysis engine is used to analyze the log samples and identify parsing efficiency issues. The code optimizer is used to generate optimized code based on the parsing efficiency issues and deploy new parsing programs based on the optimized code. The feature library manager is used to update the feature database based on the parsing efficiency issues.
8. A method for intelligent log parsing based on a three-program separation architecture, characterized in that, Log parsing is performed using the intelligent log parsing system based on a three-program separation architecture as described in any one of claims 1 to 7.