A message queue-based method for monitoring Nginx logs

Abstract

This invention proposes a method for monitoring Nginx metrics using message queues. It belongs to the field of web performance monitoring technology. The method includes: deploying Filebeat as a log collector on the Nginx server to capture Nginx access logs and error logs in real time; encapsulating the collected log data into messages using Filebeat and pushing them to a Kafka message queue via TCP protocol; a consumer pulling log messages from Kafka and parsing and processing the Nginx access logs within the messages; and aggregating Request-URL pairs with the same business meaning using an intelligent aggregation algorithm, summarizing them into a set of monitoring metrics. By capturing Nginx logs in real time using Filebeat and quickly pushing them to Kafka, the monitoring system can respond promptly to changes in the logs.

Description

Technical Field

[0001] This invention proposes an nginx log monitoring method based on message queues, belonging to the field of web performance monitoring technology. Background Technology

[0002] With the widespread use of internet applications, the performance and stability of web services have become core concerns in system operation and maintenance. Nginx, as one of the mainstream web servers, is widely used in high-concurrency and high-performance network environments. To ensure the stable operation of Nginx, log monitoring has become particularly important. By collecting and analyzing Nginx access logs, error logs, and other data in real time to generate RED (Redundant Relationship) metrics, system performance problems and potential risks can be identified promptly. Summary of the Invention

[0003] This invention provides a message queue-based nginx log monitoring method to address the issues of traditional Nginx-exporter failing to properly handle log breakpoints when log files are truncated or the Nginx service restarts, resulting in some logs not being collected; community-based nginx-exporter based on nginx log files typically runs on a single machine, making it difficult to scale. When the log volume is large, the processing power of a single node may become a bottleneck. Furthermore, since Nginx is generally composed of multiple machines, community-based nginx-exporter lacks distributed capabilities. Additionally, community-based nginx-exporter requires continuous reading of log files, and when the log volume is large, frequent file I / O operations cause high disk and CPU overhead, impacting overall performance.

[0004] This invention proposes a message queue-based nginx log monitoring method, the method comprising:

[0005] S1. Use Filebeat as a log collector and deploy it on the Nginx server to capture Nginx access logs and error logs in real time.

[0006] S2, Filebeat encapsulates the collected log data into messages and pushes them to the Kafka message queue via the TCP protocol;

[0007] S3. The consumer pulls log messages from Kafka and parses and processes the Nginx access logs in the messages;

[0008] S4. Through intelligent aggregation algorithms, request URLs with the same business meaning are aggregated and summarized into a limited set of monitoring metrics.

[0009] S5: Through its built-in rules engine and machine learning algorithms, consumers further analyze log data, extract key performance indicators, and update them to the monitoring system in real time.

[0010] S6. Generate various real-time monitoring indicators based on business needs and system configuration, and display them in the monitoring system; and implement a regular monitoring indicator expiration handling mechanism to automatically clean up monitoring indicators that have not been updated within a certain period of time.

[0011] Furthermore, S1 includes:

[0012] S11. Configure the Nginx server, install Filebeat on the Nginx server, configure Filebeat, and monitor the access logs and error logs of Nginx through Filebeat.

[0013] S12. Configure the Filebeat input module, specify the log file path, and set the buffer and backoff strategies.

[0014] S13. Configure the Filebeat output module and send log data to Kafka;

[0015] S14. Start the Filebeat service to capture new entries in the Nginx log file in real time and perform preliminary data cleaning in Filebeat.

[0016] S15. Mark the logs according to their types and add metadata, including timestamps and server identifiers;

[0017] S16. Parse the log entries using regular expressions or a JSON parser to extract key information, including IP address, request URL, and status code.

[0018] Furthermore, S2 includes:

[0019] S21. Encapsulate the parsed log data into Kafka messages and compress the Kafka messages using a compression algorithm;

[0020] S22. Configure Filebeat to push encapsulated messages to the Kafka cluster using the TCP protocol;

[0021] S23. During the push process, the log data is safely resent through an error retry mechanism when there are network fluctuations or the Kafka cluster is temporarily unavailable.

[0022] S24. Monitor the push process and record key indicators, including push success rate and latency.

[0023] Furthermore, S21 includes:

[0024] Extract key fields from the log data parsed by Filebeat. These key fields include timestamp, log level, client IP, request method, request-URL, status code, request size, and response time.

[0025] According to the Kafka message format requirements, the key fields are structured and stored as key-value pairs or JSON format, metadata is added to each structured log data, and the completeness and accuracy of the metadata are verified.

[0026] Use the Kafka client library to create Kafka message objects; encapsulate structured log data and its metadata into the Kafka message body, set the message topic and partition key; and configure the message properties;

[0027] The Kafka message body is compressed using a compression algorithm, and the compression effect is evaluated.

[0028] Furthermore, S23 includes:

[0029] During the process of pushing log data to the Kafka cluster, the status code and response information of the push operation are monitored in real time, and errors are captured and identified according to the exception handling mechanism provided by the Kafka client library.

[0030] Classify errors to determine whether they are retryable or non-retryable;

[0031] If the error is retryable, it will be retried based on the preset retry strategy, and the waiting time will be increased between each retry based on the exponential backoff strategy.

[0032] Before retrying, perform integrity and consistency checks on the log data to be resent. By leveraging Kafka's idempotency or transactional properties, data duplication or inconsistency can be avoided after network partitioning or Kafka cluster failure recovery.

[0033] During the retransmission process, record detailed information for each retry. For successful retransmissions, update the relevant status information and notify the relevant systems or personnel.

[0034] If the maximum number of retries or the total retry time limit is reached and the attempt is still unsuccessful, the failure log data will be recorded in the error log and an alarm will be triggered to notify relevant personnel to intervene and handle the situation.

[0035] Analyze the reasons for retry failures and adjust the retry strategy or optimize the system configuration based on the analysis results.

[0036] Furthermore, S3 includes:

[0037] S31. Deploy the Kafka consumer service, configure consumer groups, subscription topics and partition allocation strategies, and perform automatic consumer offset management.

[0038] S32. The consumer pulls log messages from Kafka, decompresses them, parses each message, and recovers the original Nginx log data and its metadata.

[0039] S33. Based on the log type, distribute the parsed data to different processing pipelines.

[0040] Furthermore, S33 includes:

[0041] Pull and parse log messages from Kafka, obtain the metadata or specific fields in the content of each message and identify the log type, and create a log type enumeration or dictionary;

[0042] Based on business needs and technical architecture, define multiple processing pipelines, each responsible for processing specific types of log data;

[0043] Configure the inputs, outputs, processing logic, and required resources for each processing pipeline;

[0044] Write data distribution logic to route parsed data to the corresponding processing pipeline based on log type; and use a load balancing mechanism to evenly distribute data to each processing pipeline under high concurrency and process the data asynchronously.

[0045] Establish a communication mechanism between pipelines to share and communicate data between pipelines through message queues, event buses, or distributed caches;

[0046] After each processing pipeline completes data processing, it generates monitoring metrics in the monitoring platform format and reports them to the monitoring platform.

[0047] An alarm mechanism is introduced to automatically trigger alarms and notify relevant personnel to intervene when pipeline abnormalities occur or performance indicators exceed thresholds.

[0048] Furthermore, S4 includes:

[0049] S41. On the Kafka consumer side, receive Nginx access log data from the Kafka message queue; parse each log data into structured information;

[0050] S42. Parse the Request-URL in each log and extract the key parts, including the path and query parameters;

[0051] S43. Remove irrelevant parts from the URL and standardize the URL into a comparable format;

[0052] S44. Based on preset business rules or regular expressions, further categorize standardized URLs into different business categories; generate aggregation keys based on the categorized business categories and standardized URL paths.

[0053] S45. For each aggregation key, count the number of log entries corresponding to it as the basic data for this monitoring metric; and further count the number of occurrences of different HTTP status codes under each aggregation key.

[0054] S46. Set time windows, aggregate the aggregation keys within each time window to generate time series data; dynamically adjust the aggregation granularity according to the number of monitoring indicators and the system load.

[0055] S47. Regularly clean up aggregated data that has not been updated for a long time, convert the aggregated data into monitoring metrics, and output the monitoring metrics to the monitoring system or storage system.

[0056] Furthermore, S5 includes:

[0057] S51. Configure the rule engine, define rules for identifying abnormal logs, integrate machine learning models, and automatically discover potential performance problems or abnormal patterns.

[0058] S52. Calculate key performance indicators in real time, including request response time, access volume and error rate, and update them to the monitoring system.

[0059] S53. Key performance indicators are displayed through the monitoring interface, and based on the alarm mechanism, when KPIs exceed the preset threshold, an alarm is automatically triggered to notify relevant personnel.

[0060] Furthermore, S6 includes:

[0061] S61. Based on business needs, dynamically generate real-time monitoring metrics, customize monitoring items and thresholds, and update monitoring data in real time.

[0062] S62. Set the validity period of monitoring indicators, automatically clean up expired and unupdated indicators, and optimize the cleanup strategy based on the data access frequency.

[0063] S63. Monitor the performance of the entire log monitoring system and optimize the system based on the monitoring results.

[0064] The beneficial effects of this invention are as follows: It captures Nginx logs in real time using Filebeat and quickly pushes them to Kafka, ensuring the monitoring system can respond promptly to changes in the logs; leveraging Kafka's high throughput, it can efficiently process large amounts of log data while reducing network transmission load through compression algorithms; through intelligent aggregation algorithms, request URLs with the same business meaning are grouped into a set of monitoring metrics, simplifying the complexity of monitoring data and improving data analysis efficiency; the built-in rule engine and machine learning algorithms can deeply analyze log data, extract key performance indicators, and help to promptly identify and resolve potential performance issues; and it can be adapted to business needs and system configurations. It generates and displays various real-time monitoring metrics, making monitoring results more intuitive and easier to understand; a regular monitoring metric expiration handling mechanism ensures the data accuracy and system performance of the monitoring system; during log data push, error retry mechanisms and idempotency or transactional characteristics ensure data integrity and consistency; the method supports defining multiple processing pipelines according to business needs and technical architecture, improving the system's scalability and flexibility; through performance monitoring and tuning of the entire log monitoring system, system performance can be continuously optimized and processing efficiency improved; an alarm mechanism is introduced to promptly notify relevant personnel for handling when monitoring metrics exceed preset thresholds or when abnormalities occur in the processing pipeline. Attached Figure Description

[0065] Figure 1 This is a diagram illustrating the steps of the method described in this invention. Detailed Implementation

[0066] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0067] One embodiment of the present invention, such as Figure 1 As shown, an nginx log monitoring method based on message queues is described, the method comprising:

[0068] S1. Use Filebeat as a log collector and deploy it on the Nginx server to capture Nginx access logs and error logs in real time.

[0069] S2, Filebeat encapsulates the collected log data into messages and pushes them to the Kafka message queue via the TCP protocol;

[0070] S3. Consumers (such as custom Kafka consumer services) pull log messages from Kafka and parse and process the Nginx access logs in the messages;

[0071] S4. Through intelligent aggregation algorithms, request URLs with the same business meaning are aggregated and summarized into a limited set of monitoring metrics.

[0072] S5: Through its built-in rule engine and machine learning algorithms, consumers further analyze log data, extract key performance indicators (such as request response time, access volume, error rate, etc.), and update them to the monitoring system in real time.

[0073] S6. Generate various real-time monitoring indicators based on business needs and system configuration, and display them in the monitoring system; and implement a periodic monitoring indicator expiration handling mechanism to automatically clean up monitoring indicators that have not been updated within a certain period of time (such as N hours).

[0074] The working principle of the above technical solution is as follows: Filebeat, a lightweight log collection tool, is deployed on the Nginx server. It can monitor and collect access logs (such as access.log) and error logs (such as error.log) generated by Nginx in real time. Filebeat captures newly added log entries in real time by reading the contents of the log files. Once a new log entry is detected, Filebeat processes it immediately. Filebeat encapsulates the captured log entries into a message format for subsequent transmission and processing. Each message typically contains key information such as the log timestamp, log level, and log content. The encapsulated log messages are pushed to the Kafka message queue via TCP. Kafka, as a high-performance message middleware, ensures high availability and scalability of messages. Kafka's consumer service (which can be a custom or existing log processing service) periodically pulls log messages from Kafka. These messages are managed in an ordered manner according to Kafka's partitioning and offset mechanism; the consumer service parses the pulled log messages, especially the Nginx access logs, extracting key information such as request methods, URLs, response times, and status codes; through a specific intelligent aggregation algorithm, request-URLs with the same business meaning are aggregated. This means that for similar requests, the system can treat them as the same type of monitoring object, thereby simplifying monitoring logic and reducing data redundancy; the aggregated data is summarized into a limited set of monitoring metrics, which cover various aspects of business concern, such as request volume, response time distribution, and error rate; the built-in rule engine and machine learning algorithms are used to perform in-depth analysis of the log data. The rule engine is used to identify potential anomalies or violations, while the machine learning algorithms can discover hidden patterns or trends in the data; key performance indicators (KPIs), such as request response time, access volume, and error rate, are extracted from the log data and updated to the monitoring system in real time; according to business needs and system configuration, various real-time monitoring metrics are generated and displayed in the monitoring system. These metrics are presented in the form of charts, dashboards, etc., to help operations and maintenance personnel quickly understand the system status. To prevent monitoring data from becoming outdated or consuming too much storage space, the system implements a regular monitoring metric expiration handling mechanism. Monitoring metrics that have not been updated within a certain period of time (such as N hours) are automatically cleaned up to maintain the timeliness and accuracy of monitoring data.

[0075] The effects of the above technical solution are as follows: Filebeat can capture Nginx access logs and error logs in real time, ensuring data freshness and timeliness; through the Kafka message queue, log data is pushed to the processing system in a highly efficient manner, reducing the risk of data transmission latency and loss; the Kafka consumer service can quickly pull and process log messages from the queue, achieving rapid parsing and aggregation of log data; components such as Filebeat and Kafka can be flexibly deployed on the Nginx server that needs monitoring, or deployed as an independent log processing cluster; Kafka supports horizontal scaling, and as the log volume increases, more Kafka nodes can be easily added to process more log data; the consumer service can be customized according to specific needs, flexibly processing log data and extracting the required monitoring metrics; through intelligent aggregation algorithms, requests with the same business meaning are aggregated. st-URL aggregation simplifies monitoring logic and reduces data redundancy. The built-in rule engine and machine learning algorithms enable in-depth analysis of log data, uncovering potential problems and trends, and improving monitoring accuracy and intelligence. The monitoring system displays various monitoring metrics in real time, helping operations personnel quickly understand system status. When monitoring metrics exceed preset thresholds, the system automatically triggers alarms, promptly notifying relevant personnel for handling. A regular monitoring metric expiration handling mechanism cleans up outdated data, maintaining the timeliness and accuracy of monitoring data. Through data aggregation and intelligent analysis, the amount of data requiring storage is reduced, optimizing storage resource utilization. This method can be easily integrated with existing monitoring and alarm systems to achieve unified management and monitoring of log data. As business grows, the monitoring scope can be easily expanded, monitoring metrics added, or processing logic optimized to meet ever-changing monitoring needs.

[0076] In one embodiment of the present invention, S1 includes:

[0077] S11. Configure the Nginx server, including log file path, format (such as combinedlogformat) and permission settings. Install Filebeat on the Nginx server, configure Filebeat, and monitor the Nginx access log (access.log) and error log (error.log) through Filebeat.

[0078] S12. Configure the Filebeat input module, specify the log file path, and set the buffer and backoff strategies.

[0079] S13. Configure the Filebeat output module and send log data to Kafka, including Kafka cluster address, authentication information, etc.

[0080] S14. Start the Filebeat service to capture new entries in the Nginx log file in real time, and perform preliminary data cleaning in Filebeat, such as removing blank lines and filtering invalid logs.

[0081] S15. Mark the logs according to their type (access or error) and add metadata, including timestamps and server identifiers.

[0082] S16. Parse the log entries using regular expressions or a JSON parser (if the log format is JSON) to extract key information, including IP address, request URL, and status code.

[0083] The above technical solution works as follows: Configure the Nginx server to ensure the correct paths, formats (e.g., combinedlogformat), and permission settings for log files (such as access.log and error.log). These configurations determine how Nginx generates and stores logs, and what information is recorded. Install Filebeat on the Nginx server; this is a key component for log collection. After installation, perform necessary configurations to specify how Filebeat interacts with Nginx. Configure Filebeat's input module, specifying the paths to Nginx's access and error log files. This allows Filebeat to know where to read log data. Set appropriate buffer and backoff policies to optimize the log reading process. Buffers are used to temporarily store log data before sending it to the output, while the backoff policy defines how Filebeat retryes sending data in case of network failure or temporary unavailability of the output. Configure Filebeat's output module, specifying the Kafka cluster address and necessary authentication information (if the Kafka cluster has security authentication enabled). In this way, Filebeat knows which Kafka topic to send data to. After starting the Filebeat service, it monitors new entries in the Nginx log files in real time. Whenever a new log entry is generated, Filebeat immediately captures it. While capturing log entries, Filebeat performs preliminary data cleaning operations, such as removing blank lines and filtering invalid logs, to reduce unnecessary data transmission and processing burden. Based on the content of the log entries (such as the log file name or specific markers in the log format), Filebeat types the captured log entries (access log or error log). Metadata is added to each log entry, such as a timestamp (recording the time the log entry was generated) and a server identifier (identifying the Nginx server that generated the log). This metadata helps with subsequent log processing and analysis. Depending on the Nginx log format (such as combinedlogformat or JSON format), Filebeat uses regular expressions or a JSON parser to parse the log entries. This process extracts key information from the log entries, such as IP addresses, request URLs, and status codes. This extracted key information will be used for subsequent log processing, aggregation, analysis, and monitoring. For example, IP addresses and request URLs can be used to identify different user requests and business operations; status codes can be used to assess the success rate or error rate of requests.

[0084] The effects of the above technical solution are as follows: Filebeat can capture new entries in Nginx's access and error logs in real time, ensuring data timeliness and accuracy. This is crucial for timely detection and response to system problems; by optimizing buffer and backoff strategies, Filebeat can efficiently process log data and send it to the Kafka message queue, reducing the risk of data transmission latency and loss; Filebeat's input and output modules support flexible configuration options, allowing for customization based on the specific format of Nginx logs and the configuration of the Kafka cluster; as the amount of Nginx logs increases, the capacity of the Kafka cluster can be easily expanded to handle more log data. Simultaneously, Filebeat itself supports multi-instance deployment to further improve log collection efficiency; before sending log data to Kafka, Filebeat performs preliminary data cleaning, such as removing blank lines and filtering invalid logs. This helps reduce data noise during subsequent processing and improves data quality. Filebeat adds metadata such as timestamps and server identifiers to each log entry, which is crucial for subsequent data analysis and monitoring. Using regular expressions or a JSON parser (depending on the log format), Filebeat can accurately parse log entries and extract key information such as IP addresses, request URLs, and status codes. This information is essential for understanding user behavior, monitoring system performance, and locating problems. The extracted key information provides a rich data source for subsequent log analysis, enabling operations personnel to gain a deeper understanding of system status and user behavior. Sending Nginx log data to Kafka in real time provides a solid foundation for subsequent monitoring and alerting systems. Operations personnel can build real-time monitoring metrics and alarm rules based on this data to promptly identify and respond to system problems. Because the data is captured and transmitted in real time, alarms can be triggered quickly when system problems occur, helping operations personnel to quickly locate and resolve issues. Filebeat and Kafka are both widely used open-source tools that support multiple log formats and message queue protocols, making them easy to integrate with existing monitoring and alarm systems. As business grows, the monitoring scope of Filebeat or the cluster size of Kafka can be easily expanded to meet ever-changing monitoring needs.

[0085] In one embodiment of the present invention, S2 includes:

[0086] S21. Encapsulate the parsed log data into Kafka messages and compress the Kafka messages using a compression algorithm;

[0087] S22. Configure Filebeat to push encapsulated messages to the Kafka cluster using the TCP protocol;

[0088] S23. During the push process, the log data is safely resent through an error retry mechanism when there are network fluctuations or the Kafka cluster is temporarily unavailable.

[0089] S24. Monitor the push process and record key indicators, including push success rate and latency.

[0090] The working principle of the above technical solution is as follows: Filebeat parsed log data (including complete log information and its metadata, such as timestamps, server identifiers, IP addresses, request URLs, and status codes) is encapsulated into Kafka messages. Each Kafka message should contain sufficient information to reconstruct the original log entry in subsequent processing. To reduce network transmission burden and improve efficiency, compression algorithms (such as Gzip, Snappy, etc.) are used to compress Kafka messages. Compressed data consumes less bandwidth during transmission, thereby speeding up transmission and reducing latency. Filebeat is configured to use the TCP protocol to push the encapsulated messages to the Kafka cluster. The native Kafka protocol typically has lower latency and higher throughput, making it the preferred choice when possible. Filebeat interacts with the Kafka cluster through its built-in Kafka output plugin, pushing messages to specified Kafka topics. The Kafka cluster is responsible for receiving, storing, and distributing these messages. During the push process, if abnormal situations such as network fluctuations or temporary unavailability of the Kafka cluster occur, Filebeat will trigger an error retry mechanism. This means it will attempt to resend failed log data until it succeeds or reaches a preset retry limit. This mechanism ensures the security and integrity of log data; during retries, Filebeat ensures that successfully pushed log data is not resent. This is typically achieved by tracking the status of sent messages or leveraging Kafka's idempotency and transactional properties; real-time monitoring of the Filebeat log data push process to the Kafka cluster is also crucial. This includes monitoring network status, the health of the Kafka cluster, and Filebeat performance metrics; recording and storing key metrics during the push process, such as push success rate (the ratio of successfully pushed messages to the total number of messages) and latency (the time difference between receiving log data from Filebeat and successfully pushing it to the Kafka cluster). These metrics are essential for evaluating push performance, diagnosing problems, and optimizing system configuration.

[0091] The effects of the above technical solution are as follows: By compressing Kafka messages, the amount of data transmitted over the network can be significantly reduced, thereby lowering network bandwidth consumption and transmission costs. This is particularly important for the transmission of large-scale log data; compressed data requires less time during transmission, thus accelerating the push speed of log data and improving the overall system processing efficiency; the TCP protocol can be selected for message push based on actual conditions. This flexibility allows the technical method to adapt to different network environments and Kafka cluster configurations; using the native Kafka protocol typically achieves lower latency and higher throughput, further improving system performance; in the event of network fluctuations or temporary unavailability of the Kafka cluster, the error retry mechanism ensures the safe retransmission of log data. This avoids the risk of data loss and guarantees the reliability of data transmission; during the retry process, by avoiding the retransmission of successfully pushed log data, data consistency and integrity are ensured; real-time monitoring of the push process allows for timely detection and response to potential problems, such as network failures and Kafka cluster anomalies. This helps operations personnel quickly locate problems and take corresponding measures; recording and storing key indicators such as push success rate and latency provides an important basis for performance tuning. Analyzing these metrics allows for the identification of system bottlenecks and optimization of related configurations, thereby further improving system performance. Each sub-step (such as data encapsulation, compression, push, and monitoring) is relatively independent and modular, making the system easy to expand and maintain. When new features need to be added or existing features need to be optimized, the corresponding modules can be easily modified or replaced. Using Kafka as the storage and distribution platform for log data enables seamless integration with other Kafka-based applications or tools. This reduces the system's integration costs and complexity.

[0092] In one embodiment of the present invention, step S21 includes:

[0093] Extract key fields from the log data parsed by Filebeat. These key fields include timestamp, log level, client IP, request method, request-URL, status code, request size, and response time.

[0094] According to the Kafka message format requirements, the key fields are structured and stored as key-value pairs or JSON format. Necessary metadata is added to each structured log data, such as log source (Nginx server identifier), log type (access log or error log), processing timestamp, etc., and the completeness and accuracy of the metadata are verified.

[0095] Use a Kafka client library (such as KafkaProducer) to create Kafka message objects; encapsulate structured log data and its metadata into the Kafka message body, set the message topic and partition key; and configure message attributes such as message priority and persistence strategy.

[0096] The Kafka message body is compressed using compression algorithms (such as GZIP, Snappy, LZ4, etc.), and the compression effect is evaluated.

[0097] The working principle of the above technical solution is as follows: Raw data is obtained from the log data parsed by Filebeat; based on the log data format (such as Nginx's combinedlogformat or a custom format), key fields are extracted, such as timestamp, log level, client IP, request method, request-URL, status code, request size, and response time; according to the requirements of the Kafka message format, the key fields are stored as key-value pairs or JSON format. Key-value pairs are simple and direct, suitable for simple data models; while JSON format is more flexible and can easily represent complex data structures; necessary metadata is added to each structured log data entry, such as log source (Nginx server identifier), log type (access log or error log), and processing timestamp. This metadata helps to quickly locate and identify log entries when storing and querying log data in Kafka; the added metadata is verified for completeness and accuracy to ensure correct identification and use in message transmission and subsequent processing; Kafka message objects are created using a Kafka client library (such as KafkaProducer); and the structured log data and its metadata are encapsulated into the Kafka message body. This includes setting the message topic, which determines which partition in the Kafka cluster the message will be sent to; setting the partition key, which determines where the message is stored within the partition; configuring other message attributes as needed, such as message priority (although Kafka itself does not directly support priority, it can be achieved through other means, such as using different topics or consumer groups to handle messages of different priorities), persistence strategies (such as setting message replication factors and synchronization strategies); selecting an appropriate compression algorithm (such as GZIP, Snappy, LZ4, etc.) to compress the Kafka message body according to actual needs. These algorithms each have their advantages and disadvantages in terms of compression ratio and compression speed, and need to be selected according to the actual situation; evaluating the compressed message body to ensure that while reducing data volume, it does not significantly affect the performance and efficiency of subsequent processing. This typically involves testing and analyzing metrics such as compression ratio, decompression speed, and CPU utilization; sending the compressed Kafka message to the specified Topic and partition in the Kafka cluster; and storing and distributing the message according to the configuration so that subsequent consumers (such as log analysis systems, monitoring systems, etc.) can consume and process it.

[0098] The effects of the above technical solution are as follows: By extracting key fields and storing them in a structured format as key-value pairs or JSON, log data becomes more standardized and easier to understand. This helps reduce the complexity of data parsing and improves data processing efficiency; structured log data makes storage and querying in Kafka easier. Whether using stream processing frameworks such as Kafka Streams and Spark Streaming, or exporting data to other systems (such as Elasticsearch and Hadoop) through connectors such as Kafka Connect, log data querying and analysis can be performed more efficiently; adding necessary metadata (such as log source, log type, processing timestamp, etc.) to each structured log data enhances data traceability. This allows for quick location of specific log entries and related Nginx servers when problems occur, enabling troubleshooting and repair; verifying the integrity and accuracy of metadata ensures the correctness and reliability of data during transmission and subsequent processing. This helps reduce analytical bias and incorrect decisions caused by data errors or inconsistencies; compressing the Kafka message body using compression algorithms significantly reduces data volume, thereby reducing the burden on network transmission. This facilitates rapid log data transmission in environments with limited bandwidth or high network latency; the storage efficiency of compressed data within the Kafka cluster is also improved. Due to the reduced data volume, more log data can be stored without requiring additional storage space; when creating Kafka message objects using the Kafka client library, message attributes (such as message priority, persistence strategy, etc.) can be flexibly configured. This helps adjust message processing methods and storage strategies according to actual needs to meet different business scenarios and performance requirements; by configuring message topics and partition keys, partitioned storage and parallel processing of log data can be achieved. This helps improve system scalability and processing performance, especially when processing large-scale log data; evaluating compressed data ensures that reducing volume does not significantly impact the performance and efficiency of subsequent processing. This helps select the most suitable compression algorithm and configuration parameters to achieve the best compression effect and performance balance; by monitoring and recording key metrics (such as compression ratio, decompression speed, CPU utilization, etc.), system performance and stability can be continuously optimized. This helps to promptly identify and resolve potential problems, ensuring the system can operate stably for a long time.

[0099] In one embodiment of the present invention, step S23 includes:

[0100] During the process of pushing log data to the Kafka cluster, the status code and response information of the push operation are monitored in real time. Based on the exception handling mechanism provided by the Kafka client library, error situations are captured and identified. The error situations include network anomalies and Kafka cluster unavailability.

[0101] Errors are categorized into retryable errors (such as network fluctuations or brief service unavailability) or non-retryable errors (such as data format errors or prolonged Kafka cluster unavailability).

[0102] If the error is retryable, a retry is performed based on a preset retry strategy, which includes the number of retries, the retry interval, and data integrity verification during retry; and based on an exponential backoff strategy, a waiting time is added between each retry.

[0103] Before retrying, perform integrity and consistency checks on the log data to be resent; by leveraging Kafka's idempotency or transaction features, avoid duplicate or inconsistent data being resent after network partitioning or Kafka cluster failure recovery.

[0104] During the retransmission process, record detailed information for each retry, including retry time, number of retries, and size of the retransmitted data. For log data of successful retransmissions, update the relevant status information and notify the relevant systems or personnel.

[0105] If the maximum number of retries or the total retry time limit is reached and the attempt is still unsuccessful, the failure log data will be recorded in the error log and an alarm will be triggered to notify relevant personnel to intervene and handle the situation.

[0106] Analyze the reasons for retry failures and adjust the retry strategy or optimize the system configuration based on the analysis results.

[0107] The working principle of the above technical solution is as follows: During the process of pushing log data to the Kafka cluster, the exception handling mechanism provided by the Kafka client library is used to monitor the status codes and response information of the push operation in real time. Once an error is detected (such as network anomalies, Kafka cluster unavailability, etc.), the exception is immediately captured and the error is identified. The captured errors are classified into retryable errors and non-retryable errors. Retryable errors typically include network fluctuations, brief service unavailability, etc., which may be temporary and can be successfully resolved through retries. Non-retryable errors involve more serious problems, such as data format errors, long-term Kafka cluster unavailability, etc., which require manual intervention. For retryable errors, retries are performed based on a preset retry strategy; the retry strategy includes the number of retries, the retry interval, and data integrity verification during retries; an exponential backoff strategy is adopted, adding a waiting time between each retrieval to reduce the impact on the Kafka cluster; at the same time, a maximum number of retries and a total retry time limit are set to avoid indefinite retries that lead to resource waste. Before retrying, perform integrity and consistency checks on the log data to be retransmitted to ensure that the data has not been tampered with or corrupted. Leverage Kafka's idempotency and transaction properties to ensure that retransmitted data will not be duplicated or inconsistent after network partitioning or Kafka cluster failure recovery. During the retransmission process, record detailed information for each retry, including retry time, number of retries, and the size of the retransmitted data. For successfully retransmitted log data, update relevant status information and notify relevant systems or personnel. If retry is unsuccessful after reaching the maximum number of retries or the total retry time limit, record the failed log data in the error log. Trigger an alarm to notify relevant personnel for intervention, allowing for further analysis of the problem and remedial measures. Conduct in-depth analysis of the reasons for retry failures to identify the root cause of the error. Adjust the retry strategy or optimize system configuration based on the analysis results to reduce the likelihood of similar errors in the future.

[0108] The above technical solution achieves the following effects: real-time monitoring of push operation status codes and response information enables timely detection and capture of errors, allowing for rapid response and handling. This helps reduce system instability or data loss caused by untimely error detection. Errors are categorized into retryable and non-retryable types, with different handling strategies for each type. For retryable errors, multiple attempts are made using a pre-defined retry strategy, improving the system's fault tolerance and stability. Setting maximum retry counts and total retry time limits avoids resource waste caused by indefinite retries. This helps to rationally allocate system resources and ensure that critical business operations are not affected. An exponential backoff strategy is used to increase the waiting time between each retry, reducing frequent impacts on the Kafka cluster and improving the retry success rate. Before retrying, integrity and consistency checks are performed on the log data to be retransmitted to ensure that the data has not been tampered with or corrupted. This helps maintain data accuracy and reliability. Leveraging Kafka's idempotency or transactional characteristics, data duplication or inconsistency issues are avoided after network partitions or Kafka cluster failures. This further enhances data integrity and consistency; detailed information for each retry is recorded during the retransmission process, including retry time, number of retries, and the size of the retransmitted data. This information aids in subsequent problem diagnosis and analysis; if the maximum number of retries or the total retry time limit is reached without success, the failure log data is recorded in the error log, triggering an alarm to notify relevant personnel for intervention. This helps to promptly identify problems and take appropriate remedial measures; in-depth analysis of the reasons for retry failures identifies the root causes of errors. This helps to discover potential problems in the system and take corresponding optimization measures; adjusting the retry strategy or optimizing the system configuration based on the analysis results reduces the likelihood of similar errors occurring in the future. This continuous improvement mechanism helps to improve the overall performance and stability of the system.

[0109] In one embodiment of the present invention, S3 includes:

[0110] S31. Deploy the Kafka consumer service, configure consumer groups, subscription topics (corresponding to Nginx logs), and partition allocation strategies, and perform automatic consumer offset management.

[0111] S32. The consumer pulls log messages from Kafka, decompresses them, parses each message, and recovers the original Nginx log data and its metadata.

[0112] S33. Distribute the parsed data to different processing pipelines based on the log type (access or error).

[0113] The above technical solution works as follows: Deploy the Kafka consumer service on an appropriate server. This typically involves installing Kafka client libraries (such as Java's Kafka client library) and necessary dependencies; configuring a consumer group for each consumer service instance. A consumer group is a concept in Kafka used to achieve message load balancing and high availability. Consumers within the same consumer group share the load of subscribed topics; configure consumers to subscribe to specific Kafka topics, which correspond to Nginx logs. Simultaneously, set partitioning strategies, such as Round Robin or Range allocation, to ensure messages are evenly distributed among consumers within the group; configure consumers to automatically manage their offsets in Kafka. The offset records the message position the consumer has read; automatic offset management ensures that after a consumer restarts or recovers from a failure, it can resume reading messages from the last stopped position, thus avoiding message duplication or loss; the consumer service periodically pulls log messages from subscribed topics from Kafka. These messages may have been compressed before sending to improve transmission efficiency, so consumers need to perform decompression first. Each fetched message is then parsed to recover the original Nginx log data and its metadata. This typically involves decoding the message format (such as JSON, Protobuf, etc.) and extracting key information from the logs, such as request time, request URL, and status code. After parsing the log data, it is categorized according to log type (access logs or error logs). This is usually done by examining specific fields or formats in the logs. The categorized log data is then distributed to different processing pipelines. These pipelines can be different data processing services, analytics systems, or storage systems, depending on business needs. For example, access logs might be sent to a real-time analytics system to monitor website traffic, while error logs might be sent to a log management system for troubleshooting.

[0114] The above technical solution achieves the following results: By configuring consumer groups and automatic offset management, it ensures that messages are consumed in the order they are generated in Kafka, which is particularly important for applications requiring the maintenance of event order. Automatic offset management also prevents message loss due to consumer failures or restarts, as consumers can resume consuming messages from where they left off. Using Kafka as a message queue leverages its high availability and fault tolerance features. Even if some nodes in the Kafka cluster fail, consumers can still pull messages from other available nodes, ensuring stable system operation. Consumers within a consumer group can collaborate to share the message processing load, improving the overall processing capacity and reliability of the system. Through partitioning strategies, Kafka can distribute messages across different partitions, allowing consumers within a consumer group to process messages in these partitions in parallel, thereby increasing message throughput. Consumers can dynamically adjust their message retrieval rate based on their processing capacity, avoiding performance bottlenecks caused by excessive resource usage. Consumers can decompress and parse the retrieved log messages to recover the original Nginx log data and its metadata. This allows consumers to further process and analyze log data as needed. Parsed data is distributed to different processing pipelines based on log type (access or error), enabling flexible data routing and processing to meet the diverse data processing requirements of different business needs. Kafka's high throughput and low latency characteristics enable the S3 solution to support real-time log processing. Consumers can pull log messages from Kafka almost in real-time and process and analyze them, which is particularly important for applications requiring rapid response (such as real-time monitoring and fault early warning). Modular design allows each component to be independently expanded and maintained. For example, when more consumers need to process log messages, consumer instances can be easily added and partition allocation strategies adjusted. Kafka's cluster management capabilities also simplify system operation and maintenance, allowing administrators to more easily monitor and manage the status and performance of the Kafka cluster.

[0115] In one embodiment of the present invention, S33 includes:

[0116] The consumer pulls and parses log messages from Kafka, obtains specific fields (such as log level, log type flags, etc.) from the metadata or content of each message, identifies the log type (access log or error log), and creates a log type enumeration or dictionary;

[0117] Based on business needs and technical architecture, define multiple processing pipelines, each responsible for processing specific types of log data; for example, the access log processing pipeline may include steps such as data analysis and report generation; the error log processing pipeline may include steps such as anomaly detection and alarm notification.

[0118] Configure the inputs, outputs, processing logic, and required resources (such as computing resources, storage resources, etc.) for each processing pipeline;

[0119] Write data distribution logic to route parsed data to the corresponding processing pipeline based on log type; and use a load balancing mechanism to evenly distribute data to each processing pipeline under high concurrency to avoid overloading a single pipeline; and process the data asynchronously.

[0120] Establish a communication mechanism between pipelines to share and communicate data between pipelines through message queues, event buses, or distributed caches;

[0121] After each processing pipeline completes data processing, it generates monitoring metrics in the monitoring platform format and reports them to the monitoring platform.

[0122] An alarm mechanism is introduced to automatically trigger alarms and notify relevant personnel to intervene when pipeline abnormalities occur or performance indicators exceed thresholds.

[0123] The above technical solution works as follows: Consumers pull log messages from Kafka; they parse the metadata or content of each message to obtain specific fields (such as log level, log type flags, etc.); they identify the log type (access logs or error logs) and create a log type enumeration or dictionary to quickly match and identify log types, improving processing efficiency; multiple processing pipelines are defined according to business needs and technical architecture; each pipeline is responsible for processing a specific type of log data. For example, the access log processing pipeline may include steps such as data analysis and report generation; the error log processing pipeline may include steps such as anomaly detection and alarm notification. Configure the input, output, processing logic, and required resources for each processing pipeline; write data distribution logic to route parsed data to the corresponding processing pipeline based on log type; implement data distribution through conditional statements, message routing middleware, or stream processing frameworks; introduce a load balancing mechanism to evenly distribute data across processing pipelines under high concurrency, avoiding overload of a single pipeline; process log data asynchronously to ensure that log data distribution and processing do not block the message pulling process of Kafka consumers, improving overall processing efficiency; establish a communication mechanism between pipelines for data sharing and communication through message queues, event buses, or distributed caching; after each processing pipeline completes data processing, it transmits the processing results to the next layer of the system for use; implement result feedback through logging, database updates, API calls, etc.; monitor the performance indicators of the processing pipelines in real time, such as processing speed, latency, and error rate; introduce an alarm mechanism to automatically trigger alarms to notify relevant personnel to intervene when a processing pipeline experiences an anomaly or when performance indicators exceed thresholds.

[0124] The above technical solution achieves the following effects: By quickly matching and identifying log types (using log type enumeration or dictionaries), it reduces time consumption during data processing and improves processing efficiency; the asynchronous processing mechanism ensures that log data distribution and processing do not block the message pulling process of Kafka consumers, further enhancing the overall processing capacity and response speed of the system; defining multiple processing pipelines, each responsible for processing specific types of log data, makes the system easily scalable through this modular design. When new log types need to be processed or new processing steps are added, new processing pipelines can be easily added or the configuration of existing pipelines can be modified; the load balancing mechanism ensures that the system can evenly distribute processing tasks under high concurrency, avoiding the problem of overloading a single pipeline, thereby improving the scalability and stability of the system; detailed configuration of the input, output, processing logic, and required resources for each processing pipeline helps optimize resource usage. The system can dynamically adjust resource allocation according to actual needs, avoiding resource waste and bottlenecks; communication mechanisms between pipelines (such as message queues, event buses, etc.) allow the system to efficiently share and transfer data between different processing stages, further improving resource utilization efficiency. By defining different processing pipelines and configuring specific processing logic for each pipeline, the system can flexibly process log data according to different business needs. This flexibility helps meet the data analysis needs of different departments and teams. Accurate log type identification and processing logic ensure the accuracy of data processing and reduce the possibility of errors and misunderstandings; real-time monitoring of the performance indicators of the processing pipelines (such as processing speed, latency, error rate, etc.) helps to promptly identify and resolve potential problems, ensuring the stable operation of the system; the introduction of an alarm mechanism automatically triggers alarm notifications when anomalies occur in the processing pipeline or performance indicators exceed thresholds, which can quickly attract the attention of relevant personnel and speed up the response and resolution of problems. An efficient log processing system can generate valuable analysis reports and alarm information faster, helping business personnel better understand the system's operating status and business trends, thereby improving user experience and satisfaction. By promptly processing error logs and triggering alarm notifications, the system can more quickly restore normal operation, reduce the impact of system failures on business, and enhance user trust.

[0125] In one embodiment of the present invention, step S4 includes:

[0126] S41. On the Kafka consumer side, receive Nginx access log data from the Kafka message queue; parse each log data into structured information, including but not limited to timestamp, client IP, request scheme, request-URL, status code, request size, response time, etc.

[0127] S42. Parse the Request-URL in each log and extract the key parts, including the path and query parameters.

[0128] S43. Remove irrelevant parts of the URL (such as dynamic parameters, timestamps, etc.) and standardize the URL to a comparable format; for example, standardize http: / / example.com / product?id=123&date=2023-04-01 to http: / / example.com / product.

[0129] S44. Based on preset business rules or regular expressions, further classify the standardized URLs into different business categories; generate an aggregation key based on the classified business categories and standardized URL paths.

[0130] S45. For each aggregation key, count the number of log entries corresponding to it as the basic data for this monitoring metric; and further count the number of occurrences of different HTTP status codes under each aggregation key.

[0131] S46. Set time windows (e.g., 1 minute, 5 minutes, etc.), aggregate the aggregation keys within each time window to generate time series data; dynamically adjust the aggregation granularity according to the number of monitoring indicators and the system load.

[0132] S47. Regularly clean up aggregated data that has not been updated for a long time to prevent the number of monitoring metrics from growing indefinitely. Convert the aggregated data into monitoring metrics, such as request volume, average response time, error rate, etc., and output the monitoring metrics to the monitoring system or storage system.

[0133] The working principle of the above technical solution is as follows: The processing program deployed on the Kafka consumer is responsible for pulling Nginx access log data from the Kafka message queue in real time; parsing each log data from its original format (such as text format) into structured information. This information includes, but is not limited to, timestamps, client IP, request scheme (such as HTTP / HTTPS), request-URL, status code, request size, response time, etc. This step is the foundation for subsequent processing, enabling the log data to be understood and processed by the program in a structured way; deep parsing is performed on the request-URL in each log to extract key parts, such as the path and query parameters. These parts are crucial for understanding the context and purpose of the request; to make the URLs comparable, irrelevant parts of the URL need to be removed, such as dynamic parameters (such as user ID, session ID, etc.) and timestamps. These parts are usually noise for monitoring and aggregation analysis, and removing them helps to more accurately reflect the business meaning of the request; the processed URLs are converted into a standardized format for subsequent classification and aggregation operations; according to preset business rules or regular expressions, the standardized URLs are further classified into different business categories. This step aims to group requests with similar business meanings into the same category for easier monitoring and analysis. Based on the categorized business categories and standardized URL paths, an aggregation key is generated. The aggregation key forms the basis for subsequent aggregation operations, ensuring that request-URL pairs with the same business meaning are grouped under the same monitoring metric. For each aggregation key, the number of corresponding log entries is counted as the basic data for that monitoring metric. Furthermore, the occurrence frequency of different HTTP status codes under each aggregation key is analyzed to assess key performance indicators such as request success rate and error rate. Time windows (e.g., 1 minute, 5 minutes) are set to aggregate the aggregation keys within each time window, generating time-series data. The aggregation granularity is dynamically adjusted based on the number of monitoring metrics and system load. This helps balance the level of monitoring detail with system performance, ensuring sufficient monitoring data is provided without excessively consuming system resources. Aggregate data that has not been updated for a long time is periodically cleaned up to prevent the number of monitoring metrics from growing indefinitely. This step helps keep the monitoring system clean and efficient; it transforms the aggregated data into monitoring metrics, such as request volume, average response time, and error rate, and outputs these metrics to the monitoring or storage system. This allows business personnel and system administrators to understand the system's operational status in real time and promptly identify and resolve problems.

[0134] The above technical solution achieves the following results: Real-time reception of Nginx access log data via Kafka consumers ensures timely and real-time data processing, crucial for systems requiring rapid response and real-time monitoring. URL parsing and standardization remove irrelevant parts, making different URLs comparable. This helps to more accurately analyze user request patterns and trends, identifying potential problems or anomalies. Standardized URLs are categorized into different business categories based on preset business rules or regular expressions, generating aggregation keys. This flexible categorization and aggregation mechanism allows the system to perform customized monitoring and analysis based on different business needs. The generation of aggregation keys ensures that request-URLs with the same business meaning are grouped under the same monitoring metric, simplifying the number of monitoring metrics and improving monitoring efficiency and accuracy. Statistics on the number of log entries and the frequency of different HTTP status codes under each aggregation key enable in-depth analysis of key performance indicators such as request success rate and error rate. This is significant for optimizing system performance and improving user experience. Dynamically adjusting the aggregation granularity based on the number of monitoring metrics and system load balances the level of monitoring detail with system performance. This adaptive adjustment mechanism enables the system to maintain optimal operating conditions under different scenarios. Regularly cleaning up aggregated data that hasn't been updated for a long time prevents the unlimited growth of monitoring metrics, saving storage and computing resources. Converting aggregated data into monitoring metrics and outputting them to the monitoring or storage system allows business personnel and administrators to easily view and analyze the monitoring data. Through real-time monitoring and in-depth analysis of Nginx access log data, the system can promptly detect and handle potential problems or anomalies, improving system stability and reliability. Flexible classification and aggregation mechanisms, along with configurable monitoring metrics, make the system easier to maintain and expand, reducing maintenance costs.

[0135] In one embodiment of the present invention, S46 includes:

[0136] Based on business needs and log generation rate, set the initial time window size (e.g., every 5 minutes, every hour); introduce an adaptive algorithm to dynamically adjust the time window size according to the real-time change rate of monitored data (e.g., a surge or drop in log volume); and set an overlap strategy between time windows (e.g., 50% overlap) to prevent data loss.

[0137] Within the time window, log data is grouped according to the generated aggregation key, and the number of log entries in each group is counted as basic monitoring data;

[0138] Count the occurrence frequency of different HTTP status codes (such as 200, 404, 500, etc.) under this group to build a multi-dimensional monitoring system; use a hash table to accelerate the grouping and counting process;

[0139] Real-time monitoring of system CPU, memory, and I / O usage to assess current system load; based on system load and the number of monitoring metrics, using machine learning models (such as regression analysis and time series forecasting) to predict the optimal aggregation granularity;

[0140] The system load is evaluated using the following formula:

[0141]

[0142] and, This is the load calculation function for resource X, as follows:

[0143]

[0144] in, CPU utilization (percentage) This represents memory usage as a percentage. I / O utilization rate (percentage) This refers to CPU saturation (work queue length, etc.). This refers to memory saturation (page swapping frequency, etc.). This refers to I / O saturation (such as I / O request queue length). The system's average response time (milliseconds). This is the historical load variation coefficient. , , , , , where is the weighting coefficient of each parameter, and ; , , , , , , It is a predefined threshold used to standardize the input parameters.

[0145] Based on the prediction results, the aggregation granularity within the time window is automatically adjusted. For example, the aggregation granularity is reduced during peak load periods to increase data accuracy, and the granularity is increased during off-peak periods to reduce resource consumption.

[0146] By leveraging the characteristics of time series data (such as periodicity and trends), redundant or repetitive data can be deduplicated.

[0147] By combining business rules and statistical methods, it automatically identifies and filters outliers (such as extremely high or low log counts, abnormally frequent HTTP status code changes, etc.); and converts the processed time-series data into monitoring metrics in real time, which are then output to the monitoring system or storage system.

[0148] Regularly evaluate the performance of current time series data generation and processing (such as processing speed, resource utilization, etc.), provide feedback based on the evaluation, and form a closed-loop optimization mechanism.

[0149] The working principle of the above technical solution is as follows: Initially, a basic time window size is set based on business needs and log generation rate. An adaptive algorithm is introduced, which continuously monitors the real-time change rate of log data (such as the rate of increase or decrease in log volume). When a surge or drop in log volume is detected, the adaptive algorithm dynamically adjusts the size of the time window according to preset rules or the prediction results of machine learning models. This ensures timely data processing while avoiding wasting resources when log volume is low or causing data backlog when log volume surges. To prevent data omission, an overlap strategy between time windows is set to ensure that key data points are covered multiple times, thereby improving data integrity and reliability. Within each time window, log data is grouped according to the generated aggregation key (based on business category and URL path). Efficient data structures such as hash tables are used to accelerate the grouping process, ensuring that logs with the same business category and URL path are quickly merged into the same group.

[0150] The system counts the number of log entries within each group, serving as the basic monitoring data for that group. Simultaneously, it meticulously tracks the occurrence frequency of different HTTP status codes (such as 200, 404, 500, etc.) within each group. This approach constructs a multi-dimensional monitoring system encompassing both log count and HTTP status code distribution, providing rich data support for subsequent in-depth analysis. The system also monitors real-time CPU, memory, and I / O usage to assess current system load. Based on system load and the number of monitoring metrics, machine learning models (such as regression analysis and time series forecasting) are used to predict the optimal aggregation granularity. Finally, the aggregation granularity within the time window is automatically adjusted based on the prediction results. For example, during peak load periods, the aggregation granularity can be reduced to increase data accuracy, while during off-peak periods, the granularity can be increased to reduce resource consumption; the characteristics of time series data (such as periodicity and trends) can be used for deduplication to reduce redundant data; efficient data compression algorithms (such as differential coding and run-length encoding) can be used to compress processed data to save storage space; outliers (such as extremely high or low log counts, abnormally frequent HTTP status code changes, etc.) can be automatically identified and filtered by combining business rules and statistical methods to ensure data accuracy; processed time series data can be converted into monitoring indicators in real time and output to the monitoring system or storage system for subsequent analysis and display. The performance of current time series data generation and processing (such as processing speed, resource utilization, etc.) should be evaluated regularly, and feedback should be provided based on the evaluation results. Through this closed-loop optimization mechanism, the data processing flow can be continuously optimized to improve overall performance and efficiency.

[0151] The effects of the above technical solution are as follows: By introducing an adaptive algorithm to dynamically adjust the time window size, this technical method can flexibly respond to data processing needs in different scenarios based on the real-time change rate of monitoring data, ensuring the timeliness and accuracy of data processing; setting an overlap strategy between time windows effectively prevents data omissions, improves data integrity, and makes monitoring results more reliable; based on real-time assessment of system load and the number of monitoring indicators, machine learning models are used to predict and automatically adjust the aggregation granularity, increasing data accuracy during peak load periods and reducing resource consumption during off-peak periods, thereby achieving optimal resource allocation; by accelerating the grouping and counting process through efficient data structures such as hash tables, CPU and memory consumption are significantly reduced, improving overall processing performance; detailed statistical analysis... By analyzing the frequency of different HTTP status codes within a group, a multi-dimensional monitoring system was constructed, providing richer and more accurate data support for business analysis. The characteristics of time-series data were utilized for deduplication, and efficient data compression algorithms were employed to reduce storage space usage. Simultaneously, outliers were automatically identified and filtered using business rules and statistical methods, ensuring the accuracy and reliability of the time-series data. The processed time-series data was converted into monitoring metrics in real time and output to the monitoring or storage system, enabling business personnel to obtain key monitoring information promptly, providing strong support for rapid response and decision-making. The performance of time-series data generation and processing was regularly evaluated, and feedback and optimization were based on the evaluation results, forming a closed-loop optimization mechanism to continuously improve monitoring efficiency and effectiveness. The construction of the multi-dimensional monitoring system and the accurate and reliable time-series data provide important basis for business analysis, performance tuning, and troubleshooting, helping enterprises better understand business operations, identify potential problems, and make timely optimizations and improvements. The aforementioned complex system load assessment formula comprehensively considers the utilization and saturation of multiple key resources such as CPU, memory, and I / O, as well as system response time and historical load changes, providing a comprehensive and integrated assessment of system load. By introducing an exponential function and a saturation parameter, the formula can dynamically reflect changes in resource utilization. In particular, when resource utilization approaches or exceeds a threshold, it can significantly increase the load assessment value, thereby promptly reflecting potential performance bottlenecks. The introduction of the Historical Load Variation Coefficient (HLC) enables the formula to predict possible future load trends and provide early warnings of potential performance problems. By adjusting the weighting coefficients and thresholds, the formula can be customized according to the characteristics and needs of different systems, exhibiting high flexibility.Simultaneously, it is easily expandable, allowing for the addition of more resources and performance metrics; the formula provides quantified system load assessment results, making performance monitoring and management more precise and facilitating data-driven decision-making; by identifying and quantifying system load, administrators can more effectively allocate resources and optimize performance, improving system efficiency; early warnings when system load is too high help prevent failures and interruptions caused by system overload; by optimizing resource usage, unnecessary hardware investment is reduced, the utilization rate of existing resources is improved, and operating costs are lowered; keeping system load within a reasonable range ensures a good service experience for users, reduces response time, and improves service reliability; it provides system administrators and decision-makers with a scientific basis to help them make decisions regarding system expansion, maintenance, and upgrades.

[0152] In one embodiment of the present invention, step S5 includes:

[0153] S51. Configure the rule engine, define rules for identifying abnormal logs (such as high error rate, long response time, etc.), integrate machine learning models such as time series analysis and cluster analysis, and automatically discover potential performance problems or abnormal patterns.

[0154] S52. Calculate key performance indicators (KPIs) in real time, including request response time, access volume, and error rate, and update them to the monitoring system.

[0155] S53. Key performance indicators are displayed through the monitoring interface, and based on the alarm mechanism, when KPIs exceed the preset threshold, an alarm is automatically triggered to notify relevant personnel.

[0156] The working principle of the above technical solution is as follows: A rule engine is configured within the system, defining a series of rules for identifying abnormal logs based on business needs. These rules may involve key performance indicators such as high error rates and long response times. To improve the accuracy and efficiency of anomaly detection, the rule engine integrates machine learning models, such as time series analysis and cluster analysis. These models can automatically analyze patterns and trends in log data to discover potential performance problems or abnormal patterns. The rule engine scans log data in real time or periodically, applying the defined rules and machine learning models to identify abnormal logs. When an anomaly is detected, relevant information is recorded and further processing is prepared. The system calculates a series of key performance indicators (KPIs) in real time, including but not limited to request response time, access volume, and error rate. These indicators are important bases for evaluating system performance and stability. The calculation process may involve processing, aggregating, and statistically analyzing the raw log data to ensure accurate and reliable KPI values. The calculated KPI values ​​are updated to the monitoring system in real time. In this way, business personnel and administrators can view the system's performance indicators in real time and understand the system's operating status through the monitoring system. The monitoring system provides an intuitive interface for displaying key performance indicators (KPIs). These metrics are typically presented in the form of charts, dashboards, etc., allowing users to quickly understand the overall performance and trends of the system. The system is equipped with an alarm mechanism that automatically triggers an alarm when key performance indicators (KPIs) exceed preset thresholds. These thresholds are set according to business needs and are used to determine whether the system is in an abnormal state. Alarm notifications can be sent in various ways, such as email, SMS, and instant messaging, to ensure that relevant personnel receive alert information promptly and take appropriate action.

[0157] The effects of the above technical solution are as follows: By configuring a rule engine and integrating machine learning models, the system can automatically identify abnormal logs and potential performance problems or abnormal patterns. This automated detection reduces the need for manual intervention and improves the efficiency and accuracy of anomaly detection. The system can calculate key performance indicators (KPIs) in real time and update them to the monitoring system, enabling business personnel and administrators to understand the system's operational status immediately. This real-time capability helps to respond quickly to system problems and prevent problems from escalating or affecting user experience. The introduction of machine learning models, such as time series analysis and cluster analysis, enables the system to automatically learn and adapt to patterns and trends in log data. This intelligent analysis can more accurately identify potential problems and provide deeper insights. Based on preset thresholds and alarm mechanisms, when key performance indicators (KPIs) exceed the normal range, the system will automatically trigger alarms to notify relevant personnel. This precise alarm ensures that problems are addressed and handled in a timely manner, reducing potential losses due to delays. Through real-time monitoring and anomaly detection, the system can promptly discover and handle potential performance problems or abnormal patterns, thereby improving system stability and reliability. This helps to reduce system downtime, improve user experience, and enhance business continuity. By monitoring and analyzing key performance indicators (KPIs) in real time, system administrators can understand the system's resource usage and optimize and adjust it as needed. This helps to better utilize system resources and improve overall operational efficiency. The intuitive displays and in-depth analysis provided by the monitoring interface offer crucial decision support for business personnel and administrators. They can evaluate system performance, formulate optimization strategies, and make more informed decisions based on monitoring data. When system problems occur, automatically triggered alerts can quickly notify relevant personnel, promoting team collaboration and rapid response. This helps reduce communication costs and response time, and improves team collaboration efficiency.

[0158] In one embodiment of the present invention, step S6 includes:

[0159] S61. Based on business needs, dynamically generate real-time monitoring metrics, customize monitoring items and thresholds, and update monitoring data in real time.

[0160] S62. Set the validity period of monitoring indicators (e.g., N hours), automatically clean up expired and unupdated indicators, and optimize the cleanup strategy based on the data access frequency.

[0161] S63. Monitor the performance of the entire log monitoring system, including the load of the Kafka cluster, consumer services, etc., and optimize the system based on the monitoring results, such as adding Kafka partitions and optimizing consumer configurations.

[0162] The working principle of the above technical solution is as follows: Based on specific business needs, the system can dynamically generate real-time monitoring metrics. This means the system is no longer limited to fixed monitoring items but can flexibly adjust according to business changes; users can customize monitoring items and corresponding thresholds according to actual needs. In this way, the system can more accurately reflect the key points of business operations and issue timely alerts when problems occur; the system updates monitoring data in real time to ensure users receive the latest and most accurate information. This real-time nature is crucial for timely detection and handling of potential problems; a validity period (e.g., N hours) is set for monitoring metrics to ensure the system does not retain outdated data indefinitely. This helps save storage space and improves system performance; the system automatically cleans up expired and outdated monitoring metrics. This process is automated and requires no manual intervention, thus reducing the workload of administrators; the system also optimizes the cleanup strategy based on data access frequency. For frequently accessed data, the system may extend its retention time; while for less frequently accessed data, it will clean it up more quickly. This differentiated optimization strategy helps to further improve system efficiency and resource utilization. The system monitors the performance of the entire log monitoring system, including the load of key components such as the Kafka cluster and consumer services. By monitoring the performance metrics of these components in real time, the system can promptly identify potential performance bottlenecks or anomalies. Based on the monitoring results, the system performs in-depth analysis of the monitoring data to identify the root causes of performance problems. These analyses may involve statistical analysis, aggregation, and visualization of the monitoring data. Based on the monitoring results and analysis conclusions, the system optimizes the monitoring system. This may include increasing Kafka partitions to improve throughput and optimizing consumer configurations to balance the load. Through these optimization measures, the system can process log data more efficiently, improving overall performance and stability.

[0163] The effects of the above technical solution are as follows: It dynamically generates real-time monitoring metrics based on business needs, enabling the monitoring system to flexibly respond to business changes. This flexibility ensures that the monitoring system always aligns with business objectives, improving the effectiveness and relevance of monitoring. It allows users to customize monitoring items and thresholds, enabling the monitoring system to more accurately reflect key aspects of business operations. This personalized setting enhances the adaptability and practicality of the monitoring system. Setting the validity period of monitoring metrics and automatically cleaning up expired and outdated metrics helps save storage space and reduce unnecessary resource consumption. This resource optimization strategy helps reduce system operating costs and improve resource utilization. By optimizing the cleanup strategy based on data access frequency, it further improves the efficiency and accuracy of resource management. This intelligent management approach helps ensure that critical data is retained while reducing the storage burden of non-critical data. It monitors the performance of the entire log monitoring system, including the load of key components such as the Kafka cluster and consumer services. This comprehensive monitoring helps to promptly identify and resolve performance bottlenecks, ensuring stable system operation. Based on monitoring results, the system can be optimized, such as adding Kafka partitions and optimizing consumer configurations. These optimization measures significantly improve system processing capacity and response speed, enhancing user experience and business continuity. Real-time monitoring and automated cleanup mechanisms reduce the need for manual intervention, lowering the workload of operations and maintenance (O&M) personnel. O&M staff can focus more on resolving complex issues, improving O&M efficiency and quality. Through customized monitoring items and threshold settings, the system can more accurately identify potential problems and trigger alerts. This precise alerting mechanism helps O&M personnel respond quickly and handle issues, reducing the impact of failures on business operations. Real-time updates of monitoring data ensure users receive the latest and most accurate information. This real-time nature helps users understand the operational status and trend changes of their business in a timely manner, providing strong support for business decision-making. Through in-depth analysis and visualization of monitoring data, users can more intuitively understand the operational status and potential problems. This insight helps users formulate more scientific and reasonable business strategies and optimization solutions.

[0164] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A message queue-based Nginx log monitoring method, characterized in that, The method includes: S1. Use Filebeat as a log collector and deploy it on the Nginx server to capture Nginx access logs and error logs in real time. S2, Filebeat encapsulates the collected log data into messages and pushes them to the Kafka message queue via the TCP protocol; S3. The consumer pulls log messages from Kafka and parses and processes the Nginx access logs in the messages; S4. Through intelligent aggregation algorithms, request URLs with the same business meaning are aggregated and summarized into a set of monitoring metrics. S5: Through its built-in rules engine and machine learning algorithms, consumers further analyze log data, extract key performance indicators, and update them to the monitoring system in real time. S6. Generate various real-time monitoring indicators based on business needs and system configuration, display them in the monitoring system, and implement a regular monitoring indicator expiration handling mechanism to automatically clean up monitoring indicators that have not been updated within a certain period of time. The S3 includes: S31. Deploy the Kafka consumer service, configure consumer groups, subscription topics and partition allocation strategies, and perform automatic consumer offset management. S32. The consumer pulls log messages from Kafka, decompresses them, parses each message, and recovers the original Nginx log data and its metadata. S33. Distribute the parsed data to different processing pipelines according to the log type; S33 includes: Pull and parse log messages from Kafka, obtain the metadata or specific fields in the content of each message and identify the log type, and create a log type enumeration or dictionary; Based on business needs and technical architecture, define multiple processing pipelines, each responsible for processing specific types of log data; Configure the inputs, outputs, processing logic, and required resources for each processing pipeline; Write data distribution logic to route parsed data to the corresponding processing pipeline based on log type; and use a load balancing mechanism to evenly distribute data to each processing pipeline under high concurrency and process the data asynchronously. Establish a communication mechanism between pipelines to share and communicate data between pipelines through message queues, event buses, or distributed caches; After each processing pipeline completes data processing, it generates monitoring metrics in the monitoring platform format and reports them to the monitoring platform. An alarm mechanism is introduced to automatically trigger alarms and notify relevant personnel to intervene when pipeline abnormalities occur or performance indicators exceed thresholds.

2. The nginx log monitoring method based on message queues according to claim 1, characterized in that, S1 includes: S11. Configure the Nginx server, install Filebeat on the Nginx server, configure Filebeat, and monitor the access logs and error logs of Nginx through Filebeat. S12. Configure the Filebeat input module, specify the log file path, and set the buffer and backoff strategies. S13. Configure the Filebeat output module and send log data to Kafka; S14. Start the Filebeat service to capture new entries in the Nginx log file in real time and perform preliminary data cleaning in Filebeat. S15. Mark the logs according to their types and add metadata, including timestamps and server identifiers; S16. Parse the log entries using regular expressions or a JSON parser to extract key information, including IP address, request URL, and status code.

3. The nginx log monitoring method based on message queues according to claim 1, characterized in that, The S2 includes: S21. Encapsulate the parsed log data into Kafka messages and compress the Kafka messages using a compression algorithm; S22. Configure Filebeat to push encapsulated messages to the Kafka cluster using the TCP protocol; S23. During the push process, the log data is safely resent through an error retry mechanism when there are network fluctuations or the Kafka cluster is temporarily unavailable. S24. Monitor the push process and record key indicators, including push success rate and latency.

4. The nginx log monitoring method based on message queues according to claim 3, characterized in that, S21 includes: Extract key fields from the log data parsed by Filebeat. These key fields include timestamp, log level, client IP, request method, request-URL, status code, request size, and response time. According to the Kafka message format requirements, the key fields are structured and stored as key-value pairs or JSON format, metadata is added to each structured log data, and the completeness and accuracy of the metadata are verified. Use the Kafka client library to create Kafka message objects; encapsulate structured log data and its metadata into the Kafka message body, set the message topic and partition key; and configure the message properties; The Kafka message body is compressed using a compression algorithm, and the compression effect is evaluated.

5. The nginx log monitoring method based on message queues according to claim 3, characterized in that, S23 includes: During the process of pushing log data to the Kafka cluster, the status code and response information of the push operation are monitored in real time, and errors are captured and identified according to the exception handling mechanism provided by the Kafka client library. Classify errors to determine whether they are retryable or non-retryable; If the error is retryable, it will be retried based on the preset retry strategy, and the waiting time will be increased between each retry based on the exponential backoff strategy. Before retrying, perform integrity and consistency checks on the log data to be resent. By leveraging Kafka's idempotency or transactional properties, data duplication or inconsistency can be avoided after network partitioning or Kafka cluster failure recovery. During the retransmission process, record detailed information for each retry. For successful retransmissions, update the relevant status information and notify the relevant systems or personnel. If the maximum number of retries or the total retry time limit is reached and the attempt is still unsuccessful, the failure log data will be recorded in the error log and an alarm will be triggered to notify relevant personnel to intervene and handle the situation. Analyze the reasons for retry failures and adjust the retry strategy or optimize the system configuration based on the analysis results.

6. The nginx log monitoring method based on message queues according to claim 1, characterized in that, The S4 includes: S41. On the Kafka consumer side, receive Nginx access log data from the Kafka message queue; parse each log data into structured information; S42. Parse the Request-URL in each log and extract the key parts, including the path and query parameters; S43. Remove irrelevant parts from the URL and standardize the URL into a comparable format; S44. Based on preset business rules or regular expressions, further classify standardized URLs into different business categories; generate aggregation keys based on the classified business categories and standardized URL paths; S45. For each aggregation key, count the number of log entries corresponding to it as the basic data for this monitoring metric; and further count the number of occurrences of different HTTP status codes under each aggregation key. S46. Set time windows, aggregate the aggregation keys within each time window to generate time series data; dynamically adjust the aggregation granularity according to the number of monitoring indicators and the system load. S47. Regularly clean up aggregated data that has not been updated for a long time, convert the aggregated data into monitoring metrics, and output the monitoring metrics to the monitoring system or storage system.

7. The nginx log monitoring method based on message queues according to claim 1, characterized in that, The S5 includes: S51. Configure the rule engine, define rules for identifying abnormal logs, integrate machine learning models, and automatically discover potential performance problems or abnormal patterns. S52. Calculate key performance indicators in real time, including request response time, access volume and error rate, and update them to the monitoring system. S53. Key performance indicators are displayed through the monitoring interface, and based on the alarm mechanism, when KPIs exceed the preset threshold, an alarm is automatically triggered to notify relevant personnel.

8. The nginx log monitoring method based on message queues according to claim 1, characterized in that, The S6 includes: S61. Based on business needs, dynamically generate real-time monitoring metrics, customize monitoring items and thresholds, and update monitoring data in real time. S62. Set the validity period of monitoring indicators, automatically clean up expired and unupdated indicators, and optimize the cleanup strategy based on the data access frequency. S63. Monitor the performance of the entire log monitoring system and optimize the system based on the monitoring results.

Images