Log data management method, management system construction method, equipment, medium and program product

Abstract

The invention relates to a log data management method, a management system construction method, equipment, a medium and a program product. Comprising the steps of collecting log data from various data sources through a plug-in log collection agent, processing the log data and writing the processed log data into an annular buffer area; reading target log data with a preset size from the annular buffer area, and sending the target log data to a Kafka system for storage; transmitting the data into a distributed table of a ClickHouse cluster in real time through a Kafka engine table and a materialized view; and carrying out analysis processing on the log data continuously written in the ClickHouse cluster based on the log analysis rule, generating monitoring index data, synchronizing the monitoring index data to the time sequence database, and carrying out monitoring alarm based on the alarm rule. By means of the hierarchical architecture and modular design, a management link from log collection to real-time monitoring is constructed, integrated log management which is high in performance, low in cost, easy to expand and real-time is achieved, and the requirements of enterprise-level log management and monitoring can be met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of log monitoring, in particular to a log data management method, a log data management system construction method, a computer device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] With the acceleration of enterprise digitization process, application systems are increasingly complex and distributed, and the log data generated by operation and maintenance safety and business observation presents exponential growth. In the face of huge log throughput, efficient real-time collection, low-cost storage, fast query analysis and second-level monitoring and alarm have become the core challenges to ensure system stability and drive business decision-making. The current mainstream log management technology has the following systematic bottlenecks when dealing with ultra-large-scale and high real-time requirement scenarios:

[0003] 1. High resource occupation and complex management at the collection end, making it difficult to achieve lightweight and unified collection control. Traditional collectors represented by Logstash need to rely on external storage systems (such as Elasticsearch), which have complex architecture and have problems such as configuration management difficulties, high resource occupation, etc. in large-scale distributed environments.

[0004] 2. The storage and computing layer faces the dilemma of write performance, query speed and storage cost. Taking the Elasticsearch system as an example, although it supports full-text search and real-time analysis, it has write performance bottlenecks in ultra-large-scale data scenarios, high storage costs, and complex query delays that cannot meet real-time requirements. Traditional relational databases cannot efficiently handle the write and query of massive log data, the storage structure is not suitable for unstructured logs, and the scalability is poor.

[0005] 3. The monitoring and alarm link is fragmented, with poor real-time performance, and cannot directly generate indicators based on logs and provide second-level feedback. Traditional monitoring and alarm tools centered on Prometheus have problems such as low resource utilization efficiency, insufficient data model and collection flexibility, and insufficient query performance. And Prometheus single instance is limited by memory and storage, and cannot support horizontal expansion and high availability and load balancing through sharding and replication mechanisms. SUMMARY

[0006] Therefore, it is necessary to provide a log data management method, a log data management system construction method, a computer device, a computer readable storage medium and a computer program product to construct a management link from log collection to real-time monitoring, realize integrated log management with high performance, low cost, easy expansion and real-time performance, and meet the needs of enterprise-level log management and monitoring.

[0007] In a first aspect, the application provides a log data management method, comprising:

[0008] Collecting log data from multiple data sources through a plug-in log collection agent, processing the collected log data, and writing the processed log data into a ring buffer;

[0009] Reading target log data of a preset size from the ring buffer, and sending the target log data to a Kafka system for storage;

[0010] Real-time transmission of data in the Kafka system into a ClickHouse cluster through a Kafka engine table, and writing the transmitted data into a distributed table in the ClickHouse cluster using a materialized view;

[0011] Based on a preset log analysis rule, analyzing and processing log data continuously written in the ClickHouse cluster to generate monitoring index data, synchronizing the monitoring index data to a time series database, and monitoring and alarming based on a predefined alarm rule.

[0012] In one embodiment, the method further comprises:

[0013] In response to a log query request, querying the distributed table in the ClickHouse cluster based on the query conditions in the log query request and returning the results.

[0014] In one embodiment, the plug-in log collection agent includes an input plug-in, a parser, a filter, a buffer, an output plug-in, and a ring buffer; the plug-in log collection agent collects log data from multiple data sources, processes the collected log data, and writes the processed log data into the ring buffer, comprising:

[0015] Collecting log data from multiple data sources through the input plug-in;

[0016] Converting the log data into structured data through the parser;

[0017] Filtering the structured data through the filter;

[0018] Writing the processed log data into the ring buffer through the buffer;

[0019] Reading target log data of a preset size from the ring buffer, and sending the target log data to a Kafka system for storage, comprising:

[0020] Reading target log data of a preset size from the ring buffer through the output plug-in, and sending the target log data to a Kafka system for storage.

[0021] In one of the embodiments, the Kafka system adopts a partition strategy, and distributes the target log data to different topic partitions according to the tags corresponding to the target log data; the ClickHouse cluster adopts a multi-shard architecture, and the number of shards of the ClickHouse cluster is an integer multiple of the number of partitions of the Kafka system.

[0022] In one of the embodiments, the Kafka system stores compressed log data.

[0023] In one of the embodiments, based on the preset log analysis rule, the log data continuously written in the ClickHouse cluster is analyzed and processed to generate monitoring index data, including:

[0024] Through the pre-created materialized view, based on the preset log analysis rule, the log data continuously written in the ClickHouse cluster is analyzed and processed to generate monitoring index data.

[0025] In one of the embodiments, the method further includes:

[0026] In response to the log analysis rule submitted by the user, a corresponding materialized view creation SQL statement is generated;

[0027] The materialized view creation SQL statement is executed in the ClickHouse cluster to create a target table and a corresponding materialized view; the target table is used to store the aggregated monitoring index data, and the materialized view is configured to analyze and process the log data continuously written in the ClickHouse cluster based on the log analysis rule, and refresh the monitoring index data in the target table.

[0028] In one of the embodiments, the alarm rule is pre-defined using the PromQL syntax; wherein when the monitoring index data in the time series database meets the condition of the alarm rule, an alarm event is automatically generated; according to the alarm event, a notification of monitoring alarm is sent through at least one notification channel.

[0029] In one of the embodiments, the method further includes:

[0030] In response to the configuration instruction of the collection agent, the collection rule of the plug-in log collection agent corresponding to the configuration instruction is updated.

[0031] In one of the embodiments, the method further includes:

[0032] Receive the running state information reported by each plug-in log collection agent;

[0033] Based on the running state information of each plug-in log collection agent, at least one visual display is provided in the management view.

[0034] In one embodiment, the method further comprises:

[0035] In response to the monitoring query request, initiating the monitoring query request to the time series database through a query interface exposed by the time series database based on the visualization monitoring tool;

[0036] Generating a monitoring chart according to the data returned by the time series database and conforming to the parsing format of the visualization monitoring tool;

[0037] Displaying the monitoring chart.

[0038] In a second aspect, the application further provides a log data management system construction method, the method comprising:

[0039] Building a ClickHouse cluster, a Kafka system and a time series database;

[0040] Creating a distributed table, a materialized view and a Kafka engine table, and starting a metric transmission module; the metric transmission module is used to synchronize monitoring metric data corresponding to log data in the ClickHouse cluster to the time series database;

[0041] Deploying a plug-in log collection agent on at least one remote device, and starting the plug-in log collection agent; the plug-in log collection agent is used to collect log data, and send the log data to the Kafka system for storage after processing;

[0042] Verifying a log data transmission link by initiating a log query request to the ClickHouse cluster;

[0043] Starting a monitoring center service; the monitoring center service is used to provide a log query and aggregation analysis interface to the outside;

[0044] Submitting a log analysis rule and an alarm rule through a configuration interface of the monitoring center service; in response to the log analysis rule, a corresponding materialized view is created in the ClickHouse cluster, which is used to calculate monitoring metric data in real time; based on the alarm rule, the time series database is configured, so that the time series database performs monitoring alarm when the monitoring metric data meets the condition of the alarm rule.

[0045] In a third aspect, the application further provides a computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the method of the first aspect when executing the computer program.

[0046] In a fourth aspect, the present application also provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the steps of the method according to the first aspect.

[0047] In a fifth aspect, the present application also provides a computer program product, comprising a computer program which, when executed by a processor, implements the steps of the method according to the first aspect.

[0048] The log data management method, the log data management system construction method, the computer device, the computer readable storage medium and the computer program product have the following advantages. The log data is collected from multiple data sources by the plug-in log collection agent, the collected log data is processed, and the processed log data is written into the ring buffer. The target log data of a preset size is read from the ring buffer, and the target log data is sent to the Kafka system for storage. The data in the Kafka system is transmitted into the ClickHouse cluster in real time through the Kafka engine table, and the imported data is written into the distributed table of the ClickHouse cluster by using the materialized view. The continuously written log data in the ClickHouse cluster is analyzed and processed based on the preset log analysis rule, the monitoring index data is generated, the monitoring index data is synchronized to the time series database, and monitoring alarm is performed based on the predefined alarm rule. In this way, the plug-in log collection agent is used to simplify the collection process, the ring buffer is used to realize memory recycling, and the resource occupancy rate is reduced. The Kafka system is used to realize decoupling of collection and storage, and the columnar storage and distributed architecture of the ClickHouse cluster are used to realize efficient writing of massive logs. The ClickHouse cluster and the time series database are synchronized to directly generate indexes based on logs and perform real-time alarm. The hierarchical architecture and modular design are used to build a management link from log collection to real-time monitoring, realize integrated log management with high performance, low cost, easy expansion and real-time performance, and meet the needs of enterprise-level log management and monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without creative labor.

[0050] Figure 1 A flowchart of a log data management method in an embodiment;

[0051] Figure 2Fig. 1 is a schematic diagram of an overall architecture in an embodiment;

[0052] Figure 3 Fig. 2 is a schematic diagram of relationships between data tables in an embodiment;

[0053] Figure 4 Fig. 3 is a schematic diagram of a structure of a plug-in log collection agent in an embodiment;

[0054] Figure 5 Fig. 4 is a schematic diagram of a structure of an index transmission module in an embodiment;

[0055] Figure 6 Fig. 5 is a schematic diagram of an architecture of a monitoring and alarming center in an embodiment;

[0056] Figure 7 Fig. 6 is a schematic diagram of a flow of a construction method of a log data management system in an embodiment;

[0057] Figure 8 Fig. 7 is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0058] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0059] It should be noted that the terms "first", "second", and the like used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "multiple" used in the present application refers to two or more. The term "and / or" used in the present application refers to one of the options or any combination of multiple options.

[0060] In order to facilitate understanding of the embodiments provided by the present application, some terms appearing in the specific description process are introduced as follows:

[0061] ClickHouse: is an open source columnar online analytical processing (OLAP) database management system, which adopts a distributed architecture design and supports real-time analysis and efficient query of massive data. The distributed architecture and horizontal expansion capability of ClickHouse are suitable for processing large-scale log, index and event data.

[0062] VictoriaMetrics: A high-performance time-series database compatible with Prometheus Query Language (PromQL), supporting real-time metric computation and alerting, suitable for monitoring scenarios.

[0063] Kafka: A distributed message queue system providing high-throughput and low-latency data pipeline, used in the present embodiment to decouple log collection and storage.

[0064] LogAgent: A lightweight and plugin-based log collection agent in the present embodiment, implemented in Go language, supporting cross-platform deployment with low resource consumption.

[0065] TTL (Time-To-Live): Data lifecycle policy for automatic cleanup of expired logs, optimizing storage resources.

[0066] PromQL: Prometheus Query Language used in the present embodiment to define monitoring metrics and alerting rules.

[0067] In an exemplary embodiment, as shown in Figure 1 a log data management method is provided. The method is applied to a server, which can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The method includes:

[0068] Step 102: Collecting log data from multiple data sources through a plugin-based log collection agent, processing the collected log data, and writing the processed log data into a ring buffer.

[0069] In the present embodiment, the plugin-based log collection agent (LogAgent) is implemented based on Golang and designed as a program tool for unified log collection, processing, and forwarding. It supports multiple input sources and output targets and can efficiently process structured and semi-structured data. It is configured to be suitable for various systems such as Linux, Windows, macOS, etc. The plugin-based log collection agent is deployed on each remote device that needs to collect logs. Optionally, the plugin-based log collection agent pre-allocates memory to implement the ring buffer, with memory consumption <50MB / instance.

[0070] Optionally, the plug-in log collection agent supports the following functions: supports multiple data sources (such as log files, HTTP, Syslog, etc.); provides a flexible way to process log data through plug-ins; provides a cache and retry mechanism to ensure that data is not lost; supports multiple output targets (such as ES, Kafka, HTTP, etc.); developed based on Golang, high performance, out-of-the-box, supports multiple platforms; uses Tag (tag) to identify data flow, supports data forwarding between log collection agents, and improves data sending flexibility. In this way, a lightweight LogAgent with low resource occupation (<50MB / instance) is realized based on Golang, simplifying the collection process, reducing resource occupation, supporting breakpoint resume and dynamic configuration hot loading. Through memory pre-allocation ring buffer, the GC (Garbage Collector, garbage collection) frequency is reduced.

[0071] In a specific implementation, the plug-in log collection agent collects log data through various plug-ins, and forwards the log data to the Kafka system after analysis and filtering.

[0072] Step 104, read the target log data of a preset size from the ring buffer, and send the target log data to the Kafka system for storage.

[0073] The ring buffer is a ring-shaped storage space connected at the head and tail, and the buffer is composed of fixed-size continuous memory blocks, using a first-in, first-out (FIFO) data structure. The preset size refers to a batch data volume standard that the plug-in log collection agent reads data from the ring buffer and packages and sends to Kafka, which can be: data volume size (for example, cumulative 5MB), log number (for example, cumulative 50), etc. In a specific implementation, the target log data of a preset size is read from the ring buffer and sent to the Kafka system as a batch of data for storage.

[0074] For example, referring to Figure 2 The overall architecture includes a log collection layer, a storage and computing layer, a monitoring and alarm layer, and a visualization application layer. The log collection layer includes a plug-in log collection agent and a Kafka system, and the plug-in log collection agent is deployed on different remote devices. Each plug-in log collection agent sends the collected data to the Kafka system for storage.

[0075] Step 106, the data in the Kafka system is transmitted into the ClickHouse cluster in real time through the Kafka engine table, and the imported data is written into the distributed table of the ClickHouse cluster by using the materialized view.

[0076] The ClickHouse cluster provides the ability of log data storage and query. The columnar storage and distributed table are adopted to realize horizontal expansion, the MergeTree engine is used to guarantee the data read-write efficiency, and the TTL strategy is configured to automatically clean up expired logs.

[0077] The relationship between various data tables Figure 3 Exemplarily, the local data table refers to a storage table of log data, and Table 1 shows the creation pseudo code of the local data table in an example.

[0078] Table 1:

[0079]

[0080] Exemplarily, the distributed table is used to associate the local data table and provides the ability of distributed storage and query, and Table 2 shows the creation pseudo code of the distributed table in an example.

[0081] Table 2:

[0082]

[0083] Exemplarily, the Kafka engine table refers to a data table for providing consumption of Kafka messages, and Table 3 shows the creation pseudo code of the Kafka engine table in an example.

[0084] Table 3:

[0085]

[0086] Exemplarily, the materialized view is used to provide the ability of writing data of the Kafka engine table into the distributed table, and Table 4 shows the creation pseudo code of the materialized view in an example.

[0087] Table 4:

[0088]

[0089] It can be understood that, based on the columnar storage and distributed architecture of Clickhouse, combined with Kafka, efficient writing of massive logs and sub-second query can be realized.

[0090] In step 108, based on the preset log analysis rule, the log data continuously written in the ClickHouse cluster is analyzed and processed to generate monitoring index data, the monitoring index data is synchronized to the time series database, and monitoring alarm is performed based on the pre-defined alarm rule.

[0091] The complex query result (i.e., monitoring index data) is written into a monitoring index table based on log analysis rules such as log splitting, keyword filtering, and aggregation query, and a materialized view is created. Referring to Table 5, Table 5 shows a pseudo code for creating a monitoring index table in an example.

[0092] Table 5:

[0093]

[0094] The time series database uses a VictoriaMetrics cluster to provide storage and query capabilities for log monitoring data. Specifically, referring to Figure 2 , a metric transmission module (which can be referred to as CH2VM) developed based on Golang queries the monitoring index table of the ClickHouse cluster, and outputs the monitoring index data to the VictoriaMetrics cluster through the interface protocol of metrics. Based on the predefined query syntax, a dynamic alarm rule is defined, and an alarm is triggered. The alarm information is pushed to social media platforms, emails, or short messages through the monitoring center. In this embodiment, Clickhouse and VictoriaMetrics are deeply integrated based on Golang, supporting real-time alarm based on PromQL, with low trigger delay, which can improve the real-time performance of monitoring alarm.

[0095] Exemplarily, referring to Figure 2 , the overall architecture includes a log collection layer, a storage and computing layer, a monitoring and alarm layer, and a visualization application layer. The log collection layer includes a plug-in log collection agent and a Kafka system, the storage and computing layer includes a ClickHouse cluster, the monitoring and alarm layer includes a VictoriaMetrics cluster (i.e., the time series database of this embodiment), and the visualization application layer includes a monitoring and alarm center. Real-time log data collection is achieved through LogAgent, and memory usage is reduced; storage costs are reduced using ClickHouse columnar compression, and TTL mechanism can be used to automatically clean up expired data; fast and multi-dimensional log analysis is supported, such as error distribution, request tracking, etc.; second-level alarm is realized based on VictoriaMetrics, supporting complex rules; horizontal expansion is supported through Clickhouse sharding, replication mechanism, and VictoriaMetrics cluster architecture, thereby realizing high availability and load balancing.

[0096] In the log data management method, the log data is collected from multiple data sources by the plug-in log collection agent, the collected log data is processed, and the processed log data is written into the ring buffer; the target log data of a preset size is read from the ring buffer, and the target log data is sent to the Kafka system for storage; the data in the Kafka system is transmitted into the ClickHouse cluster in real time through the Kafka engine table, and the imported data is written into the distributed table of the ClickHouse cluster by using the materialized view; based on the preset log analysis rule, the continuously written log data in the ClickHouse cluster is analyzed and processed to generate monitoring index data, the monitoring index data is synchronized to the time series database, and monitoring alarm is performed based on the predefined alarm rule. In the above manner, the plug-in log collection agent is used to simplify the collection process, the ring buffer is used to realize memory recycling, and the resource occupancy rate is reduced. The Kafka system is used to realize decoupling of collection and storage, and based on the columnar storage and distributed architecture of the ClickHouse cluster, efficient writing of massive logs can be realized. Through the synchronization of the ClickHouse cluster and the time series database, the index can be directly generated based on the log and real-time alarm can be performed. Through the hierarchical architecture and modular design, a management link from log collection to real-time monitoring is constructed, integrated log management with high performance, low cost, easy expansion and real-time performance is realized, and the demand of enterprise-level log management and monitoring can be met.

[0097] In an exemplary embodiment, the method further comprises: in response to a log query request, querying the distributed table in the ClickHouse cluster based on the query condition in the log query request and returning the result.

[0098] The ClickHouse cluster provides the ability of log data storage and query, and refer to Figure 2 The user sends a log query request to the monitoring alarm center through the reverse proxy module (Nginx), the monitoring alarm center queries the corresponding result from the ClickHouse cluster based on the query condition in the log query request, and displays the result to the user through a visual page.

[0099] In an exemplary embodiment, the plug-in log collection agent includes an input plug-in, a parser, a filter, a buffer, an output plug-in, and a ring buffer; the log data is collected from multiple data sources by the plug-in log collection agent, and step 102 includes: collecting log data from multiple data sources by the input plug-in; converting the log data into structured data by the parser; filtering the structured data by the filter; and writing the processed log data into the ring buffer by the buffer;

[0100] Step 104 includes reading the target log data of a preset size from the ring buffer through the output plug-in, and sending the target log data to the Kafka system for storage.

[0101] Wherein, referring to Figure 4 , the input plug-in (Input Plugin) is responsible for collecting data from various data sources, and the supported input sources include files, HTTP, Syslog, etc. The parser (Parser Plugin) is used to parse raw data into structured JSON format, supporting regular expressions. The filter (Filter Plugin) is used to process and convert data, filter data under certain conditions, such as adding, deleting or modifying fields. The buffer (Buffer Plugin) is used to temporarily store data to ensure reliable transmission, supporting memory buffering or file buffering, providing a retry mechanism to prevent data loss. The output plug-in (Output Plugin) is used to send processed data to the target system, supporting ES, Kafka, HTTP, etc. The ring buffer (RingBuffer) is also called a circular buffer, which is a fixed-size first-in-first-out (FIFO) data structure. Its core feature is a ring-shaped storage space connected at both ends, and the buffer is composed of fixed-size continuous memory blocks. By pre-allocating the ring buffer, frequent memory allocation and garbage collection are avoided.

[0102] In an exemplary embodiment, the Kafka system adopts a partitioning strategy, and the target log data is distributed to different topic partitions according to the tags corresponding to the target log data. The ClickHouse cluster adopts a multi-shard architecture, and the number of shards of the ClickHouse cluster is an integer multiple of the number of partitions of the Kafka system.

[0103] Wherein, the Kafka system adopts a partitioning strategy, and data is divided into different topic partitions (Topic) according to tags (Tag), improving parallelism. The number of shards of the ClickHouse cluster is an integer multiple (assuming N times) of the number of partitions of the Kafka system. One partition of the Kafka system corresponds to N shards of the ClickHouse cluster. In the process of writing data from the Kafka system to the ClickHouse cluster, data distribution can be based on the correspondence between partitions and shards to improve processing efficiency.

[0104] In an exemplary embodiment, the Kafka system stores compressed log data.

[0105] Among them, the Kafka system adopts GZIP for data compression, and the received log data is stored after compression, which can reduce the network transmission overhead. Further, the Kafka system adopts a replica mechanism, and sets the replica factor to 3 to ensure high availability of data.

[0106] In an exemplary embodiment, based on the preset log analysis rule, the log data continuously written in the ClickHouse cluster is analyzed and processed to generate monitoring index data, including: based on the preset log analysis rule, the log data continuously written in the ClickHouse cluster is analyzed and processed through the pre-created materialized view to generate monitoring index data.

[0107] Among them, the materialized view is a database object that persistently stores the results of a query as a physical table and automatically refreshes with source data updates. Unlike ordinary views (virtual tables), it actually occupies storage space, but the query performance is extremely high. When the system is initialized or the rule is configured, the structure and calculation logic of the materialized view are defined in advance according to the log analysis rule, which serves as part of the data pipeline and is ready before the log data arrives, ensuring real-time processing capability. The preset log analysis rule refers to the rule set defined by the user through the configuration interface or API to guide log analysis. The entire processing process constitutes an efficient real-time index calculation pipeline: log data stream continuously writes to ClickHouse source table, pre-created materialized view listens to data insertion, immediately executes analysis query according to preset rule, and writes aggregated results to index target table. This design realizes the second-level conversion from raw logs to monitoring indexes, providing real-time and accurate monitoring data sources for the system, and is the technical cornerstone to support second-level alarm and instant insight.

[0108] In an exemplary embodiment, the method further comprises: in response to the log analysis rule submitted by the user, generating a corresponding materialized view creation SQL statement; executing the materialized view creation SQL statement in the ClickHouse cluster to create a target table and a corresponding materialized view; the target table is used to store the aggregated monitoring index data, and the materialized view is configured to analyze and process the log data continuously written in the ClickHouse cluster based on the log analysis rule, and refresh the monitoring index data in the target table.

[0109] Among them, Figure 2The monitoring alarm center shown supports monitoring rule configuration, can configure log analysis rules based on SQL syntax in response to user operations, and generate corresponding materialized views. Exemplarily, the log analysis rule refers to a structured instruction set defined by a user for extracting valuable information from massive logs, which can specify: data source, filtering condition, aggregation dimension, aggregation calculation strategy, etc. The server receives user configuration, parses its logic, and fills in a predefined SQL template to obtain a materialized view creation SQL statement. The server sends and runs the SQL statement to any node in the cluster in a programmatic way through the interface provided by the ClickHouse cluster, and the ClickHouse cluster creates the target table and the materialized view on all related shards of the distributed cluster after receiving the statement. The target table serves as the output destination and persistent storage of the materialized view, and is used to store the aggregated monitoring metric data. The internal logic of the materialized view corresponds to the log analysis rule submitted by the user, and other solidified calculation logic of the rule: based on the log analysis rule, the log data continuously written in the ClickHouse cluster is analyzed and processed, and the monitoring metric data in the target table is refreshed. Thus, accurate and timely data supply is provided for second-level monitoring alarm.

[0110] In an exemplary embodiment, the alarm rule is predefined in PromQL syntax; wherein when the monitoring metric data in the time series database meets the condition of the alarm rule, an alarm event is automatically generated; and according to the alarm event, a notification of the monitoring alarm is sent through at least one notification channel.

[0111] wherein, Figure 2 The monitoring alarm center shown supports alarm rule configuration based on PromQL syntax to configure alarm rules and trigger alarm notifications, and supports notification channels such as social media platforms, SMS, email, Syslog, etc. The alarm engine of the time series database periodically executes PromQL queries, triggers alarms when the query results meet the preset conditions in the alarm rules, and sends the automatically generated alarm events to relevant management personnel through notification channels such as social media platforms, SMS, email, Syslog, etc.

[0112] In an exemplary embodiment, the method further comprises: in response to the configuration instruction of the collection agent, updating the collection rule of the plug-in log collection agent corresponding to the configuration instruction.

[0113] wherein, Figure 2The illustrated monitoring alarm center supports LogAgent management, and the collection rules of the LogAgent can be configured and updated remotely. The configuration instruction of the collection agent carries a target identifier (such as an Agent ID or a host group label), and the monitoring alarm center can accurately route the instruction to one or more specified Agent instances to achieve batch or accurate control. The monitoring alarm center generates (or is submitted to the monitoring alarm center by a user) and issues a configuration instruction, the instruction reaches the target LogAgent through the network, the LogAgent analyzes the instruction and performs configuration difference analysis, dynamically loads / unloads plug-ins, rebuilds a data processing pipeline, switches data flow atomically, applies new collection rules, reports an update success state, and the monitoring alarm center interface is refreshed in real time.

[0114] In an exemplary embodiment, the method further comprises: receiving running state information reported by each plug-inized log collection agent; and providing at least one visual display in a management view based on the running state information of each plug-inized log collection agent.

[0115] Each plug-inized log collection agent refers to a LogAgent instance distributed in each server, virtual machine or container, and each instance is an independently running process. The LogAgent actively pushes a multidimensional data set reflecting the real-time health status and performance of the LogAgent to the monitoring alarm center, i.e., running state information. The management view refers to a visual page or panel in the front-end Web interface of the monitoring alarm center for presenting the running state of the system. The monitoring alarm center visually presents the received running state information through a graphical method.

[0116] In an exemplary embodiment, the method further comprises: in response to a monitoring query request, initiating a monitoring query request to the time series database through a query interface exposed by the visual monitoring tool based on the time series database; generating a monitoring chart according to data returned by the time series database and conforming to an analysis format of the visual monitoring tool; and displaying the monitoring chart.

[0117] In an exemplary embodiment, the method further comprises: in response to a monitoring query request, initiating a monitoring query request to the time series database through a query interface exposed by the visual monitoring tool based on the time series database; generating a monitoring chart according to data returned by the time series database and conforming to an analysis format of the visual monitoring tool; and displaying the monitoring chart. Figure 2The monitoring alarm center shown supports monitoring visualization, which interfaces with Grafana monitoring charts to display monitoring data in real time. Among them, the monitoring query request refers to the retrieval instruction of specific monitoring index data initiated by the user through the visualization interface. The visualization monitoring tool refers to Grafana, which is an open-source visualization analysis platform specially used for chart display of monitoring data, supports multiple data sources, provides rich chart types (line chart, column chart, dashboard, etc.) and dashboard customization functions. Grafana sends monitoring query requests to the VictoriaMetrics cluster through the query interface exposed by the VictoriaMetrics cluster. The VictoriaMetrics cluster returns data in a format that conforms to Grafana's parsing format after executing the query. Grafana receives and parses the data, renders the chart (draws curves, column charts, etc.) according to the chart configuration, and the chart is embedded in the dashboard and displayed in the user's browser.

[0118] In an exemplary embodiment, referring to Figure 5 , the index transmission module (which can be referred to as CH2VM) developed based on Golang includes: a source component (‌Metrics Source) for collecting and transmitting monitoring indexes, a buffer (Buffer), and a component (‌Metrics Sink) for receiving and processing monitoring index data. It queries the monitoring index table of the ClickHouse cluster, outputs the monitoring index data to the VictoriaMetrics cluster through the metrics interface protocol, and realizes data synchronization.

[0119] In an exemplary embodiment, referring to Figure 6 , the ClickHouse cluster synchronizes the monitoring index data corresponding to the log data to the VictoriaMetrics cluster through the index transmission module. The VictoriaMetrics cluster monitors the monitoring index data based on predefined alarm rules, and when the monitoring index data meets the alarm rule conditions, it performs monitoring alarm and sends alarm information through the monitoring alarm center. In response to user requests, in the case of a monitoring query request, through the Grafana visualization monitoring tool, based on the query interface exposed by the VictoriaMetrics cluster, the relevant data is queried and visualized. In the case of a log management request, the monitoring alarm center accesses the ClickHouse query API based on the query conditions in the log query request, retrieves logs or performs aggregate analysis from the ClickHouse cluster, and displays them to the user through the visualization page. The monitoring alarm center also supports proxy management of the collection rules and collection states of the plug-in log collection agent. The monitoring alarm center also supports configuring log analysis rules for the index transmission module.

[0120] In an exemplary embodiment, as shown in Figure 7 A method for constructing a log data management system is provided, which is applied to a server, which can be a single physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The method comprises:

[0121] Step 702, building a ClickHouse cluster, a Kafka system and a time series database.

[0122] Among them, the ClickHouse cluster (for example, 6 shards 3 replicas), the Kafka system (for example, 6 partitions 3 replicas) and the VictoriaMetrics cluster (3 nodes read and write at the same time) are built, and the load balancing is realized through Nginx.

[0123] Step 704, creating a distributed table, a materialized view and a Kafka engine table, and starting a metric transmission module; the metric transmission module is used to synchronize the monitoring metric data corresponding to the log data in the ClickHouse cluster to the time series database.

[0124] Among them, the ClickHouse distributed table, the materialized view, the Kafka engine table and the distributed metric data table are created, and the CH2VM is started. Referring to Table 6, Table 6 shows the creation pseudo code of the metric transmission module.

[0125] Table 6:

[0126]

[0127] Step 706, deploying a plug-in log collection agent on at least one remote device, and starting the plug-in log collection agent; the plug-in log collection agent is used to collect log data, and the log data is sent to the Kafka system for storage after processing.

[0128] Among them, LogAgent is deployed, the configuration file is configured, and LogAgent is started. Referring to Table 7 and Table 8, Table 7 shows the configuration file of LogAgent in an example, and Table 8 shows the data collected by LogAgent in an example.

[0129] Table 7:

[0130]

[0131] Table 8:

[0132]

[0133] Step 708, verifying the log data transmission link by initiating a log query request to the ClickHouse cluster.

[0134] Wherein, the Clickhouse data query function is verified, and Table 9 shows the log query result in an example.

[0135] Table 9:

[0136]

[0137] Step 710, starting the monitoring center service; the monitoring center service is used to provide log query and aggregation analysis interface to the outside.

[0138] Wherein, the monitoring center service is started to realize Figure 2 The monitoring management center shown in the figure provides the functions of query service to retrieve logs and perform aggregation analysis.

[0139] Step 712, submitting log analysis rules and alarm rules through the configuration interface of the monitoring center service; wherein, in response to the log analysis rules, corresponding materialized views are created in the ClickHouse cluster for real-time calculation of monitoring index data; based on the alarm rules, the time series database is configured, so that the time series database performs monitoring alarm when the monitoring index data meets the alarm rule condition.

[0140] Wherein, the log monitoring alarm is configured, the log analysis rules and alarm rules are created, so as to trigger the alarm based on the log data management system subsequently.

[0141] In this embodiment, a log data management system capable of managing logs and monitoring alarms is created. The system can simplify the collection process by using plug-in log collection agents, realize memory recycling through a ring buffer, and reduce resource occupancy. Through the Kafka system, decoupling of collection and storage is realized, and based on the columnar storage and distributed architecture of the ClickHouse cluster, efficient writing of massive logs can be realized. Through the synchronization of ClickHouse cluster and time series database, indicators can be directly generated based on logs and real-time alarms can be performed. Through hierarchical architecture and modular design, a management link from log collection to real-time monitoring is built, realizing integrated log management with high performance, low cost, easy expansion and real-time performance, which can meet the needs of enterprise-level log management and monitoring.

[0142] It should be understood that although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless explicitly stated herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowchart involved in each embodiment as described above can include multiple steps or multiple stages, which are not necessarily executed at the same time but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential but can be alternately or alternately executed with at least some of the other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0143] In an exemplary embodiment, a computer device, which can be a server, has an internal structure diagram as shown in Figure 8 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a log data management method or a construction method of a log data management system.

[0144] Those skilled in the art can understand that Figure 8 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0145] In one example embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program: collecting log data from multiple data sources through a plug-in log collection agent, processing the collected log data, and writing the processed log data into a ring buffer; reading target log data of a preset size from the ring buffer, and sending the target log data to a Kafka system for storage; through a Kafka engine table, real-time transmission of data in the Kafka system into a ClickHouse cluster, and using a materialized view, writing the transmitted data into a distributed table of the ClickHouse cluster; based on a preset log analysis rule, analyzing and processing log data continuously written in the ClickHouse cluster to generate monitoring index data, synchronizing the monitoring index data to a time series database, and performing monitoring alarm based on a predefined alarm rule.

[0146] In one embodiment, the processor further implements the following steps when executing the computer program: in response to a log query request, querying the distributed table in the ClickHouse cluster based on query conditions in the log query request and returning a result.

[0147] In one embodiment, the processor further implements the following steps when executing the computer program: collecting log data from multiple data sources through an input plug-in; converting the log data into structured data through a parser; filtering the structured data through a filter; writing the processed log data into a ring buffer through a buffer; reading target log data of a preset size from the ring buffer through an output plug-in, and sending the target log data to a Kafka system for storage.

[0148] In one embodiment, the processor further implements the following steps when executing the computer program: through a pre-created materialized view, based on a preset log analysis rule, analyzing and processing log data continuously written in the ClickHouse cluster to generate monitoring index data.

[0149] In one embodiment, the processor further implements the following steps when executing the computer program: in response to a log analysis rule submitted by a user, generating a corresponding materialized view creation SQL statement; executing the materialized view creation SQL statement in the ClickHouse cluster to create a target table and a corresponding materialized view; the target table is used to store aggregated monitoring index data, and the materialized view is configured to analyze and process log data continuously written in the ClickHouse cluster based on the log analysis rule, and refresh the monitoring index data in the target table.

[0150] In one embodiment, the processor, when executing the computer program, further implements the following steps: in response to the configuration instruction of the collection agent, updating the collection rule of the plug-in log collection agent corresponding to the configuration instruction.

[0151] In one embodiment, the processor, when executing the computer program, further implements the following steps: receiving the running state information reported by each plug-in log collection agent; and providing at least one visual display in the management view based on the running state information of each plug-in log collection agent.

[0152] In one embodiment, the processor, when executing the computer program, further implements the following steps: in response to the monitoring query request, initiating the monitoring query request to the time series database through the query interface exposed by the time series database based on the visual monitoring tool; generating the monitoring chart according to the data returned by the time series database and conforming to the analysis format of the visual monitoring tool; and displaying the monitoring chart.

[0153] In one embodiment, the processor, when executing the computer program, further implements the following steps: building a ClickHouse cluster, a Kafka system and a time series database; creating a distributed table, a materialized view and a Kafka engine table, and starting a metric transmission module; the metric transmission module is used to synchronize the monitoring metric data corresponding to the log data in the ClickHouse cluster to the time series database; deploying a plug-in log collection agent on at least one remote device and starting the plug-in log collection agent; the plug-in log collection agent is used to collect log data and send the log data to the Kafka system for storage after processing; verifying the log data transmission link by initiating a log query request to the ClickHouse cluster; starting a monitoring center service; the monitoring center service is used to provide a log query and aggregation analysis interface to the outside; submitting a log analysis rule and an alarm rule through a configuration interface of the monitoring center service; in response to the log analysis rule, a corresponding materialized view is created in the ClickHouse cluster for real-time calculation of monitoring metric data; based on the alarm rule, the time series database is configured so that the time series database performs monitoring alarm when the monitoring metric data meets the condition of the alarm rule.

[0154] In one embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the following steps: collecting log data from a plurality of data sources through a plug-in log collection agent, processing the collected log data, and writing the processed log data into a ring buffer; reading target log data of a preset size from the ring buffer, and sending the target log data to a Kafka system for storage; transmitting data in the Kafka system into a ClickHouse cluster in real time through a Kafka engine table, and using a materialized view to write the transmitted data into a distributed table of the ClickHouse cluster; based on a preset log analysis rule, analyzing and processing log data continuously written in the ClickHouse cluster to generate monitoring index data, synchronizing the monitoring index data to a time series database, and performing monitoring alarm based on a predefined alarm rule.

[0155] In one embodiment, the computer program is executed by the processor to further implement the following steps: in response to a log query request, querying a distributed table in the ClickHouse cluster based on a query condition in the log query request and returning a result.

[0156] In one embodiment, the computer program is executed by the processor to further implement the following steps: collecting log data from a plurality of data sources through an input plug-in; converting the log data into structured data through a parser; filtering the structured data through a filter; writing the processed log data into a ring buffer through a buffer; reading target log data of a preset size from the ring buffer through an output plug-in, and sending the target log data to a Kafka system for storage.

[0157] In one embodiment, the computer program is executed by the processor to further implement the following steps: based on a preset log analysis rule, analyzing and processing log data continuously written in the ClickHouse cluster through a pre-created materialized view to generate monitoring index data.

[0158] In one embodiment, the computer program is executed by the processor to further implement the following steps: in response to a log analysis rule submitted by a user, generating a corresponding materialized view creation SQL statement; executing the materialized view creation SQL statement in the ClickHouse cluster to create a target table and a corresponding materialized view; the target table is used to store aggregated monitoring index data, and the materialized view is configured to analyze and process log data continuously written in the ClickHouse cluster based on the log analysis rule to refresh the monitoring index data in the target table.

[0159] In one embodiment, the computer program, when executed by the processor, further implements the following steps: in response to the configuration instruction of the collection agent, updating the collection rule of the plug-in log collection agent corresponding to the configuration instruction.

[0160] In one embodiment, the computer program, when executed by the processor, further implements the following steps: receiving the running state information reported by each plug-in log collection agent; and providing at least one visual display in the management view based on the running state information of each plug-in log collection agent.

[0161] In one embodiment, the computer program, when executed by the processor, further implements the following steps: in response to the monitoring query request, initiating the monitoring query request to the time series database through the query interface exposed by the time series database based on the visual monitoring tool; generating the monitoring chart according to the data returned by the time series database and conforming to the analysis format of the visual monitoring tool; and displaying the monitoring chart.

[0162] In one embodiment, the computer program, when executed by the processor, further implements the following steps: building a ClickHouse cluster, a Kafka system and a time series database; creating a distributed table, a materialized view and a Kafka engine table, and starting a metric transmission module; the metric transmission module is used to synchronize the monitoring metric data corresponding to the log data in the ClickHouse cluster to the time series database; deploying a plug-in log collection agent on at least one remote device and starting the plug-in log collection agent; the plug-in log collection agent is used to collect log data and send the processed log data to the Kafka system for storage; verifying the log data transmission link by initiating a log query request to the ClickHouse cluster; starting a monitoring center service; the monitoring center service is used to provide a log query and aggregation analysis interface to the outside; submitting a log analysis rule and an alarm rule through a configuration interface of the monitoring center service; in response to the log analysis rule, creating a corresponding materialized view in the ClickHouse cluster for real-time calculation of monitoring metric data; and based on the alarm rule, configuring the time series database so that the time series database performs monitoring alarm when the monitoring metric data meets the condition of the alarm rule.

[0163] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the following steps: collecting log data from a plurality of data sources through a plug-in log collection agent, processing the collected log data, and writing the processed log data into a ring buffer; reading target log data of a preset size from the ring buffer, sending the target log data to a Kafka system for storage; through a Kafka engine table, real-time transmission of data in the Kafka system into a ClickHouse cluster, and using a materialized view, writing the transmitted data into a distributed table of the ClickHouse cluster; based on a preset log analysis rule, analyzing and processing log data continuously written in the ClickHouse cluster to generate monitoring index data, synchronizing the monitoring index data to a time series database, and performing monitoring alarm based on a predefined alarm rule.

[0164] In one embodiment, the computer program, when executed by the processor, further implements the following steps: in response to a log query request, querying the distributed table in the ClickHouse cluster based on query conditions in the log query request and returning a result.

[0165] In one embodiment, the computer program, when executed by the processor, further implements the following steps: collecting log data from a plurality of data sources through an input plug-in; converting the log data into structured data through a parser; filtering the structured data through a filter; writing the processed log data into a ring buffer through a buffer; reading target log data of a preset size from the ring buffer through an output plug-in, and sending the target log data to a Kafka system for storage.

[0166] In one embodiment, the computer program, when executed by the processor, further implements the following steps: through a pre-created materialized view, based on a preset log analysis rule, analyzing and processing log data continuously written in the ClickHouse cluster to generate monitoring index data.

[0167] In one embodiment, the computer program, when executed by the processor, further implements the following steps: in response to a log analysis rule submitted by a user, generating a corresponding materialized view creation SQL statement; executing the materialized view creation SQL statement in the ClickHouse cluster to create a target table and a corresponding materialized view; the target table is used to store aggregated monitoring index data, and the materialized view is configured to analyze and process log data continuously written in the ClickHouse cluster based on the log analysis rule, and refresh the monitoring index data in the target table.

[0168] In one embodiment, the computer program, when executed by the processor, further implements the following steps: in response to the configuration instruction of the collection agent, updating the collection rule of the plug-in log collection agent corresponding to the configuration instruction.

[0169] In one embodiment, the computer program, when executed by the processor, further implements the following steps: receiving the running state information reported by each plug-in log collection agent; and providing at least one visual display in the management view based on the running state information of each plug-in log collection agent.

[0170] In one embodiment, the computer program, when executed by the processor, further implements the following steps: in response to the monitoring query request, initiating the monitoring query request to the time series database through the query interface exposed by the time series database based on the visual monitoring tool; generating the monitoring chart according to the data returned by the time series database and conforming to the analysis format of the visual monitoring tool; and displaying the monitoring chart.

[0171] In one embodiment, the computer program, when executed by the processor, further implements the following steps: building a ClickHouse cluster, a Kafka system and a time series database; creating a distributed table, a materialized view and a Kafka engine table, and starting a metric transmission module; the metric transmission module is used to synchronize the monitoring metric data corresponding to the log data in the ClickHouse cluster to the time series database; deploying the plug-in log collection agent on at least one remote device and starting the plug-in log collection agent; the plug-in log collection agent is used to collect log data and send the processed log data to the Kafka system for storage; verifying the log data transmission link by initiating a log query request to the ClickHouse cluster; starting a monitoring center service; the monitoring center service is used to provide a log query and aggregation analysis interface to the outside; submitting a log analysis rule and an alarm rule through the configuration interface of the monitoring center service; in response to the log analysis rule, a corresponding materialized view is created in the ClickHouse cluster for real-time calculation of monitoring metric data; based on the alarm rule, the time series database is configured so that the time series database performs monitoring alarm when the monitoring metric data meets the condition of the alarm rule.

[0172] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0173] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0174] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0175] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A method for managing log data, characterized in that, The method includes: Log data is collected from multiple data sources through a pluggable log collection agent, the collected log data is processed, and the processed log data is written into a circular buffer. Read target log data of a preset size from the circular buffer and send the target log data to the Kafka system for storage; Data from the Kafka system is transmitted to the ClickHouse cluster in real time via the Kafka engine table, and materialized views are used to write the transmitted data into the distributed tables of the ClickHouse cluster. Based on preset log analysis rules, the continuously written log data in the ClickHouse cluster is analyzed and processed to generate monitoring indicator data. The monitoring indicator data is synchronized to the time series database, and monitoring and alarms are generated based on predefined alarm rules.

2. The method according to claim 1, characterized in that, The method further includes: In response to a log query request, the system queries the distributed tables in the ClickHouse cluster based on the query conditions in the log query request and returns the results.

3. The method according to claim 1, characterized in that, The pluggable log collection agent includes an input pluggable, a parser, a filter, a buffer, an output pluggable, and a circular buffer. The process of collecting log data from multiple data sources through the pluggable log collection agent, processing the collected log data, and writing the processed log data into the circular buffer includes: The input plugin collects log data from multiple data sources. The parser transforms the log data into structured data. The structured data is filtered using the filter described above; The processed log data is written to the circular buffer through the buffer. The step of reading target log data of a preset size from the circular buffer and sending the target log data to the Kafka system for storage includes: The output plugin reads target log data of a preset size from the circular buffer and sends the target log data to the Kafka system for storage.

4. The method according to claim 1, characterized in that, The Kafka system employs a partitioning strategy, distributing the target log data to different topic partitions according to the tags corresponding to the target log data; the ClickHouse cluster adopts a multi-shard architecture, and the number of shards in the ClickHouse cluster is an integer multiple of the number of partitions in the Kafka system.

5. The method according to claim 1, characterized in that, The Kafka system stores compressed log data.

6. The method according to claim 1, characterized in that, The method, based on preset log analysis rules, analyzes and processes the continuously written log data in the ClickHouse cluster to generate monitoring metric data, including: By using pre-created materialized views and based on preset log analysis rules, the continuously written log data in the ClickHouse cluster is analyzed and processed to generate monitoring metric data.

7. The method according to claim 6, characterized in that, The method further includes: In response to user-submitted log analysis rules, generate corresponding materialized view creation SQL statements; Execute the materialized view creation SQL statement in the ClickHouse cluster to create a target table and a corresponding materialized view. The target table is used to store aggregated monitoring metric data, and the materialized view is configured to analyze and process the log data continuously written to the ClickHouse cluster based on the log analysis rules, and refresh the monitoring metric data in the target table.

8. The method according to claim 1, characterized in that, The alarm rules are predefined using PromQL syntax; when the monitoring metric data in the time series database meets the conditions of the alarm rules, an alarm event is automatically generated; based on the alarm event, a monitoring alarm notification is sent through at least one notification channel.

9. The method according to claim 1, characterized in that, The method further includes: In response to the configuration command of the collection agent, the collection rules of the plug-in log collection agent corresponding to the configuration command are updated.

10. The method according to claim 1, characterized in that, The method further includes: Receive runtime status information reported by each plug-in log collection agent; The running status information of each plug-in log collection agent is provided in at least one visualization in the management view.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: In response to a monitoring query request, the monitoring query request is initiated to the time series database through a visual monitoring tool based on the query interface exposed by the time series database; Based on the data returned by the time-series database that conforms to the parsing format of the visualization monitoring tool, a monitoring chart is generated; The monitoring charts are then displayed.

12. A method for constructing a log data management system, characterized in that, The method includes: Set up a ClickHouse cluster, a Kafka system, and a time-series database; Create distributed tables, materialized views, and Kafka engine tables, and start the metric transmission module; the metric transmission module is used to synchronize the monitoring metric data corresponding to the log data in the ClickHouse cluster to the time series database; Deploy a pluggable log collection agent on at least one remote device and start the pluggable log collection agent; the pluggable log collection agent is used to collect log data, process the log data and send it to the Kafka system for storage; The log data transmission link is verified by sending a log query request to the ClickHouse cluster. Start the monitoring center service; the monitoring center service is used to provide external interfaces for log querying and aggregation analysis. Log analysis rules and alarm rules can be submitted through the configuration interface of the monitoring center service. In response to the log analysis rules, a corresponding materialized view is created in the ClickHouse cluster for real-time calculation of monitoring metric data. The time-series database is configured based on the alarm rules so that the time-series database will issue monitoring alarms when the monitoring metric data meets the conditions of the alarm rules.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images