Method and system for realizing large-scale node monitoring based on ELK Stack technology

Through ELK Stack technology, combined with Filebeat, Kafka, Logstash, Elasticsearch and Kibana, efficient monitoring and visualization of large-scale nodes is achieved, solving the pain points of traditional monitoring methods, and improving system stability and troubleshooting efficiency.

CN120582967APending Publication Date: 2025-09-02BEIJING INSPUR CLOUD COMPUTING CO LTD

Patent Information

Application Number
CN202510691950.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-27
Publication Date
2025-09-02

AI Technical Summary

Technical Problem

Traditional monitoring and log management methods cannot meet the storage and analysis needs of massive log data in large-scale systems. The log correlation analysis across nodes is complicated, making it difficult for operation and maintenance personnel to quickly locate the root cause of the problem.

Method used

Using ELK Stack technology, Filebeat is used for data acquisition, Kafka is used as message queue buffer, Logstash is used for data processing, Elasticsearch is used as core storage and search engine, and Kibana is used for visual display and alarm to realize large-scale node monitoring.

Benefits of technology

Improves system stability, reduces operation and maintenance costs, accelerates troubleshooting, supports data-driven decision-making in multiple business scenarios, and provides comprehensive operation and maintenance monitoring solutions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120582967A_ABST
    Figure CN120582967A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computers, in particular to a method and system for achieving large-scale node monitoring based on the ELK Stack technology, and the method comprises the following steps: data collection, message queue buffering, data processing, data storage, and visual display and alarm. The method has the advantages that large-scale node monitoring is achieved based on the ELK Stack technology, the problem of pain points of a traditional log management mode is solved, and a comprehensive operation and maintenance monitoring solution is provided for enterprises through the efficient data processing capacity, the flexible expansibility and the powerful visualization function of the method. The core value of the method is to improve the stability of the system, reduce the operation and maintenance cost, accelerate troubleshooting and support data-driven decisions under various business scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for implementing large-scale node monitoring based on ELK Stack technology. Background Art

[0002] With the popularization of cloud computing and microservice architecture, the scale of systems continues to expand, and the number of nodes may reach thousands or even tens of thousands.

[0003] Traditional monitoring and log management methods cannot meet the storage and analysis needs of massive log data. When a system failure occurs, cross-node log correlation analysis is complex, making it difficult for operations personnel to quickly locate the root cause of the problem. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for implementing large-scale node monitoring based on ELK Stack technology to solve the problems raised in the above background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for implementing large-scale node monitoring based on ELK Stack technology, comprising the following steps:

[0006] Data collection: Use Filebeat as a lightweight log collector and deploy it on large-scale nodes;

[0007] Message queue buffer: The data sent by Filebeat is transmitted to the Kafka cluster. The Kafka cluster acts as a buffer and decoupling layer to temporarily store and buffer the data.

[0008] Data processing: Logstash is used to receive data from the Kafka cluster. Logstash processes the data in three steps: input, filter, and output.

[0009] Data storage: Use Elasticsearch as the core storage and search engine to receive and store structured data from Logstash;

[0010] Visualization and alerting: Using Kibana as a visualization tool, we can transform the massive node data stored in Elasticsearch into an intuitive graphical interface through dynamic dashboards, geographic heat maps, and multi-dimensional analysis.

[0011] Preferably, data collection specifically includes: using the Harvesters component of Filebeat to monitor the specified log file, regularly detecting the creation, deletion, update and rename operations of the file, and when new content is found in the file, reading the new log line line by line starting from the last read position of the file, and encapsulating each line of log into an event object containing the original content of the log as well as the file path and timestamp metadata; using the Spooler component as a memory buffer to temporarily store the encoded events to be sent, and when the number of events reaches a certain threshold or a certain time interval, sending the events in batches to the specified output target, the output target including at least one of Logstash, Elasticsearch, and Kafka.

[0012] Preferably, data processing specifically includes: in the Input stage, reading data from the Kafka cluster and converting it into the internal event format of Logstash; in the Filter stage, combining and configuring the grok plug-in according to actual needs to parse the unstructured log text into structured fields, the mutate plug-in to perform basic operations and type conversion on the fields, the date plug-in to parse the timestamp in the log and set it as the event time, the geoip plug-in to parse the geographic location information according to the IP address, and the useragent plug-in to parse the user agent string; in the Output stage, sending the processed events to Elasticsearch.

[0013] Preferably, data storage specifically includes: Elasticsearch works in the form of a cluster, the cluster consists of multiple nodes, and the nodes include master nodes, data nodes and coordination nodes; the master node is responsible for managing the metadata of the cluster, the data node stores data and performs data-related operations, and the coordination node receives client requests, routes requests and summarizes the results and returns them to the client; data is stored in the form of segments in Elasticsearch, and Elasticsearch regularly performs segment merging operations to merge multiple small segments into a large segment.

[0014] Preferably, the visual display and alarm specifically include: setting up an alarm system, obtaining fault signals through threshold rules and machine learning anomaly detection strategies, and notifying relevant personnel in real time.

[0015] A system for implementing a method for large-scale node monitoring based on ELK Stack technology, comprising:

[0016] Data collection layer: Deploy Filebeat on large-scale nodes and configure Filebeat to collect node log files;

[0017] Message queue layer: Deploy a Kafka cluster and use it as a buffer and decoupling layer for data transmission. It receives data from Filebeat and temporarily stores the data to ensure the stability and reliability of data transmission.

[0018] Data processing layer: Configure Logstash to receive data from the Kafka cluster;

[0019] Data storage layer: Build an Elasticsearch cluster. Elasticsearch serves as a central data warehouse, distributed search engine, data analysis engine, and observability hub, receiving and storing structured data from Logstash.

[0020] Visualization layer and alarm system: Use Kibana for data visualization and create dynamic dashboards, geographic heat maps, and multi-dimensional analysis views to present node data in an intuitive graphical interface.

[0021] Preferably, in the data collection layer, the Harvesters component of Filebeat is responsible for monitoring changes in log files, reading new log lines in real time, and encapsulating log lines into event objects, which contain the original content and metadata of the logs; the Spooler component acts as a buffer pool to batch process the encoded events and send the batch events to the specified output target according to the preset threshold or time interval, and the output target is the Kafka cluster.

[0022] Preferably, in the data processing layer, the Input plug-in of Logstash reads data from the Kafka cluster and converts it into an internal event format; the Filter plug-in sequentially calls the grok plug-in, mutate plug-in, date plug-in, geoip plug-in and useragent plug-in according to the rules in the configuration file to process and convert the events, converting unstructured data into structured data, and standardizing, cleaning, time-unifying, geo-location parsing and user agent information parsing the data; the Output plug-in sends the processed events to Elasticsearch.

[0023] Preferably, in the data storage layer, the Elasticsearch cluster uses a sharding mechanism to achieve horizontal expansion and high availability of data. Data is stored in the form of segments, and query performance is improved through segment merging operations. The master node manages cluster metadata, the data node is responsible for data storage and operations, and the coordination node processes client requests.

[0024] Preferably, in the visualization layer and the alarm system, an alarm system is established, threshold rules and machine learning anomaly detection strategies are set, and when data anomalies occur, an alarm signal is issued in a timely manner to notify relevant personnel to conduct troubleshooting.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] This paper proposes a method and system for large-scale node monitoring based on ELK Stack technology. This approach not only addresses the pain points of traditional log management but also provides enterprises with a comprehensive operations and maintenance monitoring solution through its efficient data processing capabilities, flexible scalability, and powerful visualization capabilities. Its core value lies in improving system stability, reducing operations and maintenance costs, accelerating troubleshooting, and supporting data-driven decision-making in a variety of business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a system architecture diagram of the present invention;

[0028] Figure 2 This is the Logstash architecture diagram of the present invention;

[0029] Figure 3 This is the ES architecture diagram of the present invention. DETAILED DESCRIPTION

[0030] In order to clearly and completely describe the objectives and technical solutions of the present invention and make the advantages more clearly understood, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, not all of them, and are only used to explain the embodiments of the present invention, not to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] In the first embodiment, the present invention provides a technical solution: a method for implementing large-scale node monitoring based on ELK Stack technology, comprising the following steps:

[0032] Data collection steps: In a large-scale node environment, Filebeat is selected as the log collection tool; Filebeat's Harvesters component continuously monitors the specified log file, detects changes in the file status, and immediately reads the log line by line once new content is found, and converts each line of log into an event containing the original content and metadata; the Spooler component acts as a buffer area to batch store the encoded events. When the batch sending conditions are met, the events are sent to the Kafka cluster to ensure efficient data transmission.

[0033] Message queue transmission steps: Build a Kafka cluster as the message queue layer. The Kafka cluster receives data from Filebeat, buffers and temporarily stores the data, realizes the decoupling between the data collection layer and the data processing layer, and improves the stability and scalability of the system.

[0034] Data processing steps: Configure Logstash to receive data from the Kafka cluster; the Input stage of Logstash reads data from the Kafka cluster and converts it into internal events; in the Filter stage, based on actual business needs, flexibly combines the grok plug-in, mutate plug-in, date plug-in, geoip plug-in, and useragent plug-in to parse, convert, and process events. For example, use the grok plug-in to parse Nginx logs, the mutate plug-in to clean up redundant fields, the date plug-in to unify the time format, the geoip plug-in to parse the geographic location information of the IP address, and the useragent plug-in to parse the user agent information; the Output stage sends the processed events to Elasticsearch.

[0035] Data storage steps: Build an Elasticsearch cluster. Elasticsearch serves as the core component for data storage and search, receiving and storing structured data from Logstash. Elasticsearch adopts a distributed architecture, achieving horizontal scalability and high availability of data through sharding and replication. Data is stored in segments, and Elasticsearch periodically merges segments to reduce disk I / O and improve query efficiency. The master node is responsible for cluster management, the data node is responsible for data storage and operations, and the coordination node is responsible for request routing and result aggregation.

[0036] Visualization and alerting steps: Use Kibana for data visualization. By creating various visual charts and dashboards, the node data stored in Elasticsearch is displayed in an intuitive way, allowing users to quickly view node status. At the same time, an alert system is set up to monitor data changes in real time based on threshold rules and machine learning anomaly detection algorithms. When an anomaly is found, an alert notification is issued in a timely manner so that users can handle the problem in a timely manner.

[0037] In the second embodiment, based on the first embodiment, a system for implementing a method for large-scale node monitoring based on the ELK Stack technology is proposed, including:

[0038] (1) Data Collection Layer: Filebeat is a lightweight log collector in the Beat family. It uses few resources and is suitable for large-scale deployments. It collects log files from nodes. It consists of two core components: Harvesters and Spoolers. Harvesters monitor designated log files and regularly detect file creation, deletion, update, and rename operations. When new content is detected, the harvester reads new log lines starting from the last read position. It reads the file content line by line and processes each line as an event. After reading the log lines, the harvester encapsulates them into event objects. Each event contains the original log content as well as metadata such as the file path and timestamp. The events are then encoded into a suitable format for subsequent transmission. The spooler acts as a memory buffer, temporarily storing events to be sent. Encoded events are sent to the spooler for batch processing, improving data transmission efficiency. When the number of events in the spooler reaches a certain threshold or a certain time interval, the spooler sends these events in batches to the output destination. Filebeat supports multiple output targets, such as Logstash, Elasticsearch, Kafka, etc. According to the output section in the configuration file, Spooler will send batch events to the specified target.

[0039] (2) Message queue layer: Kafka cluster, as a buffer and decoupling layer, will not be explained in detail here.

[0040] (3) Data processing layer: Logstash receives data from Filebeat, filters, converts and processes the data, and then sends the processed data to Elasticsearch. The working principle of Logstash can be divided into three main steps: input, filter and output. The architecture diagram is as follows Figure 2 As shown in the figure. The Input plugin reads data from various data sources (such as files, databases, message queues, and log systems) and converts it into Logstash's internal event format. It acts like a "data entry point," introducing external data into the Logstash processing pipeline. The Filter plugin is a core component of the Logstash data processing flow, processing and transforming data after it enters the input source before it is sent to the output destination. The following is a detailed description of the functions of the five most commonly used Filter plugins:

[0041] (1) The core function of the grok plug-in is to parse unstructured log text into structured fields, use predefined or custom regular expression patterns to match text, support multi-level matching and splitting of complex logs, and is mostly used to parse raw logs of services such as Nginx and Apache.

[0042] (2) The core function of the mutate plugin is to perform basic operations and type conversions on fields, including renaming, deleting, and copying fields; converting between string, number, and Boolean types; converting strings to uppercase and lowercase, and merging strings; and if conditional judgments. It is often used for field format standardization and cleaning up redundant fields.

[0043] (3) The core function of the date plug-in is to parse the timestamp in the log and set it as the event time, including extracting the time from the specified field. It also supports custom time formats and can automatically convert time zones to unify the log time format and solve time zone issues.

[0044] (4) The core function of the geoip plug-in is to parse geographic location information based on the IP address, and parse out more than 20 fields such as country, city, longitude and latitude, etc. It also supports custom field mapping and is mostly used in network access analysis and security auditing.

[0045] (5) The core function of the useragent plug-in is to parse the user agent string (User-Agent) to extract information such as device, browser, and operating system. It is often used for Web access log analysis.

[0046] Logstash's filter plugins receive events from the input stage and, based on the rules in the configuration file, sequentially invoke different filter plugins to process and transform the events. They ultimately pass the processed events to the output stage. Different filter plugins have different functions and can be combined and configured to meet various data processing requirements. Finally, the processed events are passed to the output plugin, which sends them to a specified destination, such as Elasticsearch.

[0047] (IV) Data storage layer: Elasticsearch is the core storage and search engine of ELK Stack, playing the following key roles: Central data warehouse: receiving and storing structured data from Logstash / Beats; distributed search engine: providing near real-time full-text search capabilities; data analysis engine: supporting complex aggregation calculations; observability center: unified storage of logs, indicators, and application tracking data. Based on the distributed storage and search architecture of Elasticsearch, the sharding mechanism is used to achieve horizontal expansion and high availability of data. The ES architecture diagram is as follows: Figure 3 shown.

[0048] Elasticsearch works in a cluster, consisting of multiple nodes (servers), each of which can play different roles. The master node is responsible for managing the cluster's metadata, such as creating and deleting indices, and coordinating the cluster's status.

[0049] Data nodes: store data and perform data-related operations, such as adding, deleting, modifying, and querying data. Coordinating nodes: receive client requests, route them to the appropriate nodes, and aggregate the results and return them to the client.

[0050] ES data writing process:

[0051] (1) Client request: The client sends a write request to the Elasticsearch cluster, which first reaches the coordinating node.

[0052] (2) Routing calculation: The coordinating node calculates which shard the document should be stored on based on the document ID or routing rules. The index in Elasticsearch is split into multiple shards, and each shard can have multiple replicas.

[0053] (3) Primary Shard Write: The coordinating node forwards the write request to the node where the primary shard is located. After receiving the request, the primary shard writes the document to the memory buffer and records it in the transaction log.

[0054] (4) Replica synchronization: After the primary shard is successfully written, the data will be synchronized to the corresponding replica (RepliaShard) shard. Only when all replica shards are successfully written will a successful write response be returned to the coordinating node.

[0055] ES data storage mechanism:

[0056] Segments: Data in Elasticsearch is stored as segments. Segments are immutable and cannot be modified once created. New data is written to a new segment. Over time, a large number of small segments are generated. Elasticsearch periodically merges segments into a single large segment to reduce disk I / O and improve query performance.

[0057] ES data search process:

[0058] (1) Client request: The client sends a search request to the Elasticsearch cluster, which also first reaches the coordination node.

[0059] (2) Query distribution: The coordinating node forwards the query request to all relevant shards (including primary shards and replica shards).

[0060] (3) Shard query: After receiving a query request, each shard performs the query operation in the local segment and returns the ID and related parts of the document that meets the conditions.

[0061] (4) Result aggregation: The coordinating node collects the results returned by all shards, aggregates and sorts the results.

[0062] (5) Document acquisition: The coordination node obtains the complete document content from the corresponding shard based on the summarized results.

[0063] (6) Result return: The coordination node returns the final search results to the client.

[0064] Elasticsearch achieves efficient data writing, storage, and search capabilities through a distributed architecture, sharding and replication mechanisms, segment storage, and merging technologies. Its distributed nature enables it to handle large amounts of data and highly concurrent requests, making it widely used in log analysis, full-text search, data analysis, and other fields.

[0065] (5) Visualization layer and alerting system: With Kibana at its core, this layer transforms massive amounts of node data into intuitive graphical interfaces through dynamic dashboards, geographic heat maps, and multi-dimensional analysis, enabling personnel to more quickly identify abnormal conditions. The alerting system uses strategies such as threshold rules and machine learning anomaly detection to detect fault signals, ensuring that problems are proactively identified before they impact business operations.

[0066] A large-scale node monitoring solution based on the ELK Stack uses lightweight Beats agents to collect node metrics and logs, which are then buffered by Kafka and filtered by Logstash before being stored in an Elasticsearch cluster. This ensures efficient indexing and querying of large amounts of data. Kibana captures global status, and alert systems like ElastAlert provide real-time notifications. This solution supports large-scale node access through horizontal scalability and utilizes a shard replication mechanism to ensure high availability, ultimately enabling timely fault location.

[0067] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for implementing large-scale node monitoring based on ELK Stack technology, characterized by: The following steps are involved: Data collection: Use Filebeat as a lightweight log collector and deploy it on large-scale nodes; Message queue buffer: The data sent by Filebeat is transmitted to the Kafka cluster. The Kafka cluster acts as a buffer and decoupling layer to temporarily store and buffer the data. Data processing: Logstash is used to receive data from the Kafka cluster. Logstash processes the data in three steps: input, filter, and output. Data storage: Use Elasticsearch as the core storage and search engine to receive and store structured data from Logstash; Visualization and alerting: Using Kibana as a visualization tool, we can transform the massive node data stored in Elasticsearch into an intuitive graphical interface through dynamic dashboards, geographic heat maps, and multi-dimensional analysis.

2. The method for implementing large-scale node monitoring based on ELK Stack technology according to claim 1, characterized in that: Data collection specifically includes: using Filebeat's Harvesters component to monitor specified log files, regularly detecting file creation, deletion, update, and rename operations, and when new content is found in the file, reading new log lines line by line starting from the last read position of the file, and encapsulating each line of log into an event object containing the original log content as well as file path and timestamp metadata; using the Spooler component as a memory buffer to temporarily store encoded events to be sent, and when the number of events reaches a certain threshold or a certain time interval, sending the events in batches to the specified output target, which includes at least one of Logstash, Elasticsearch, and Kafka.

3. The method for implementing large-scale node monitoring based on ELK Stack technology according to claim 2, characterized in that: Data processing specifically includes: in the Input stage, reading data from the Kafka cluster and converting it into Logstash's internal event format; in the Filter stage, combining and configuring the grok plug-in according to actual needs to parse unstructured log text into structured fields, the mutate plug-in to perform basic operations and type conversion on the fields, the date plug-in to parse the timestamp in the log and set it as the event time, the geoip plug-in to parse geographic location information based on the IP address, and the useragent plug-in to parse the user agent string; in the Output stage, the processed events are sent to Elasticsearch.

4. The method for implementing large-scale node monitoring based on ELK Stack technology according to claim 3, characterized in that: Data storage specifically includes: Elasticsearch works in the form of a cluster, which consists of multiple nodes, including master nodes, data nodes and coordination nodes; the master node is responsible for managing the cluster's metadata, the data node stores data and performs data-related operations, and the coordination node receives client requests, routes requests, and summarizes the results and returns them to the client; data is stored in the form of segments in Elasticsearch, and Elasticsearch regularly performs segment merge operations to merge multiple small segments into one large segment.

5. The method for implementing large-scale node monitoring based on ELK Stack technology according to claim 4, characterized in that: Visual display and alarm specifically include: setting up an alarm system, obtaining fault signals through threshold rules and machine learning anomaly detection strategies, and notifying relevant personnel in real time.

6. A system for implementing the method for large-scale node monitoring based on ELK Stack technology according to claim 5, characterized in that: include: Data collection layer: Deploy Filebeat on large-scale nodes and configure Filebeat to collect node log files; Message queue layer: Deploy a Kafka cluster and use it as a buffer and decoupling layer for data transmission. It receives data from Filebeat and temporarily stores the data to ensure the stability and reliability of data transmission. Data processing layer: Configure Logstash to receive data from the Kafka cluster; Data storage layer: Build an Elasticsearch cluster. Elasticsearch serves as a central data warehouse, distributed search engine, data analysis engine, and observability hub, receiving and storing structured data from Logstash. Visualization layer and alarm system: Use Kibana for data visualization and create dynamic dashboards, geographic heat maps, and multi-dimensional analysis views to present node data in an intuitive graphical interface.

7. A system according to claim 6, characterized in that: In the data collection layer, Filebeat's Harvesters component is responsible for monitoring changes in log files, reading new log lines in real time, and encapsulating log lines into event objects, which contain the original content and metadata of the logs. The Spooler component acts as a buffer pool, batch-processing the encoded events and sending the batch events to the specified output target based on the preset threshold or time interval. The output target is the Kafka cluster.

8. A system according to claim 7, characterized in that: In the data processing layer, Logstash's Input plug-in reads data from the Kafka cluster and converts it into an internal event format. The Filter plug-in calls the grok plug-in, mutate plug-in, date plug-in, geoip plug-in, and useragent plug-in in sequence according to the rules in the configuration file to process and convert events, converting unstructured data into structured data, and standardizing, cleaning, time-aligning, geo-location parsing, and user agent information parsing for the data. The Output plug-in sends the processed events to Elasticsearch.

9. A system according to claim 8, characterized in that: In the data storage layer, the Elasticsearch cluster uses a sharding mechanism to achieve horizontal expansion and high availability of data. Data is stored in the form of segments, and query performance is improved through segment merging operations. The master node manages cluster metadata, the data node is responsible for data storage and operations, and the coordination node processes client requests.

10. A system according to claim 9, characterized in that: In the visualization layer and alarm system, an alarm system is established, threshold rules and machine learning anomaly detection strategies are set, and when data anomalies occur, an alarm signal is issued in a timely manner to notify relevant personnel to conduct troubleshooting.

Citation Information

Patent Citations

  • A cloud log collection method based on distributed architecture

    CN109344033A

  • Log analysis monitoring system and method based on ELK and Zabbix

    CN110309030A

  • Log processing system, log processing method, terminal and storage medium

    CN110362544A

  • Improved ELK log analysis method for cluster system

    CN111104302A

  • Data analysis method and system, electronic equipment and storage medium

    CN113485892A

Cited By

  • Visual operation and maintenance management system based on distributed cloud platform

    CN120856599A

  • Search engine cluster management method and device, electronic equipment and program product

    CN120881152A