Nginx monitoring index scheme of message queue

Abstract

The invention provides an nginx monitoring index scheme of a message queue. The method belongs to the technical field of web performance monitoring. According to the scheme, the method comprises the steps that Filebeat serves as a log collector and is deployed on an Nginx server, and access logs and error logs of the Nginx are captured in real time; the Filebeat encapsulates the collected log data into a message, and pushes the message to a Kafka message queue through a TCP (Transmission Control Protocol); the consumer pulls the log message from the Kafka, and analyzes and processes the Nginx access log in the message; and through an intelligent aggregation algorithm, performing aggregation processing on the Request-URLs with the same service meaning, and summarizing the Request-URLs as a monitoring index set. Nginx logs are captured in real time through Filebeat and quickly pushed to Kafka, and it is ensured that a monitoring system can respond to changes in the logs in time.

Description

Technical Field

[0001] The present invention proposes a monitoring metric scheme for Nginx of message queues, belonging to the technical field of web performance monitoring. Background Art

[0002] With the wide use of Internet applications, the performance and stability of Web services have become the core concerns in system operation and maintenance. As one of the mainstream Web servers, Nginx is widely used in high-concurrency and high-performance network environments. To ensure the stable operation of Nginx, log monitoring becomes particularly important. By collecting and analyzing Nginx access logs, error logs, etc. in real time, RED metrics can be generated to timely discover system performance problems and potential risks. Summary of the Invention

[0003] The present invention provides a monitoring metric scheme for Nginx of message queues to solve the problems that traditional Nginx-exporter may not be able to correctly handle log breakpoints when the log file is truncated or the Nginx service is restarted, resulting in some logs not being collected; the community's nginx-expoter based on Nginx log files usually runs in a single machine and is difficult to scale. When the log volume is huge, the processing of a single node may become a bottleneck, and Nginx is generally composed of multiple machines, and the community's nginx-exporter does not have the ability of distribution, and the community's nginx-expoter needs to continuously read log files. When the log volume is large, frequent file I / O operations will cause high disk and CPU overhead, affecting the overall performance:

[0004] A monitoring metric scheme for Nginx of message queues proposed by the present invention, the scheme includes:

[0005] S1. Use Filebeat as a log collector and deploy it on the Nginx server to capture Nginx access logs and error logs in real time;

[0006] S2. Filebeat encapsulates the collected log data into messages and pushes them to the Kafka message queue through the TCP protocol;

[0007] S3. Consumers pull log messages from Kafka and parse and process the Nginx access logs in the messages;

[0008] S4. Through an intelligent aggregation algorithm, aggregate and process Request-URLs with the same business meaning and classify them into a limited set of monitoring metrics;

[0009] S5. Through the built-in rule engine and machine learning algorithm, consumers can further analyze the log data, extract key performance indicators, and update them to the monitoring system in real time;

[0010] S6. Generate various real-time monitoring indicators based on business needs and system configuration, and display them in the monitoring system; and implement a regular monitoring indicator expiration processing mechanism to automatically clean up monitoring indicators that have not been updated within a certain period of time.

[0011] Furthermore, the S1 includes:

[0012] S11. Configure the Nginx server, install Filebeat on the Nginx server, configure Filebeat, and monitor the Nginx access log and error log through Filebeat.

[0013] S12. Configure the input module of Filebeat, specify the log file path, and set the buffer and backoff strategies;

[0014] S13. Configure the output module of Filebeat and send log data to Kafka;

[0015] S14. Start the Filebeat service to capture new entries in the Nginx log file in real time and perform preliminary data cleaning in Filebeat.

[0016] S15. Mark the log according to the log type and add metadata, where the metadata includes a timestamp and a server identifier.

[0017] S16. Parse the log entry using a regular expression or JSON parser to extract key information, including the IP address, request URL, and status code.

[0018] Furthermore, the S2 includes:

[0019] S21. Encapsulate the parsed log data into a Kafka message and compress the Kafka message using a compression algorithm;

[0020] S22. Configure Filebeat to push the encapsulated message to the Kafka cluster using the TCP protocol.

[0021] During the push process, the error retry mechanism ensures that log data is safely resent when the network fluctuates or the Kafka cluster is temporarily unavailable.

[0022] S24: Monitor the push process and record key indicators, including push success rate and delay.

[0023] Furthermore, the S21 includes:

[0024] Extract key fields from the log data parsed by Filebeat, including timestamp, log level, client IP, request method, Request-URL, status code, request size, and response time;

[0025] According to the Kafka message format requirements, the key fields are structured and stored as key-value pairs or JSON format, and metadata is added to each structured log data; and the integrity and accuracy of the metadata are verified;

[0026] Use the Kafka client library to create a Kafka message object; encapsulate structured log data and its metadata into the Kafka message body, set the message topic and partition key; and configure the message properties;

[0027] Compress the Kafka message body using a compression algorithm and evaluate the compression effect.

[0028] Furthermore, the S23 includes:

[0029] When pushing log data to the Kafka cluster, the status code and response information of the push operation are monitored in real time. Errors are captured and identified using the exception handling mechanism provided by the Kafka client library.

[0030] Classify errors to determine whether they are retryable or non-retryable errors;

[0031] If it is a retryable error, it will be retried based on the preset retry strategy, and based on the exponential backoff strategy, the waiting time will be increased between each retry;

[0032] Before retrying, the integrity and consistency of the log data to be resent are checked. By using Kafka's idempotence or transactional features, duplicate or inconsistent data can be avoided after network partition or Kafka cluster failure recovery.

[0033] During the resending process, the detailed information of each retry is recorded. For the successfully resent log data, the relevant status information is updated and the relevant systems or personnel are notified.

[0034] If the maximum number of retries or the total retry time limit is reached and the attempt is still unsuccessful, the failed log data will be recorded in the error log and an alarm will be triggered to notify relevant personnel to intervene;

[0035] Analyze the reasons for retry failures and adjust the retry strategy or optimize the system configuration based on the analysis results.

[0036] Furthermore, the S3 includes:

[0037] S31. Deploy the Kafka consumer service, configure the consumer group, subscription topic, and partition allocation strategy, and perform automatic consumer offset management.

[0038] S32: The consumer pulls log messages from Kafka, decompresses them, parses each message, and recovers the original Nginx log data and its metadata.

[0039] S33. Distribute the parsed data to different processing pipelines based on the log type

[0040] Furthermore, the S33 includes:

[0041] Pull and parse log messages from Kafka, obtain specific fields in the metadata or content of each message and identify the log type, and create a log type enumeration or dictionary;

[0042] Based on business needs and technical architecture, define multiple processing pipelines, each responsible for processing a specific type of log data;

[0043] Configure the input, output, processing logic, and required resources of each processing pipeline;

[0044] Write data distribution logic to route parsed data to the corresponding processing pipeline based on log type. Use load balancing to evenly distribute data to various processing pipelines in high-concurrency situations and process the data asynchronously.

[0045] Establish a communication mechanism between pipelines to share and communicate data between pipelines through message queues, event buses, or distributed caches;

[0046] After completing data processing, each processing pipeline generates monitoring indicators in the format of the monitoring platform and reports them to the monitoring platform.

[0047] An alarm mechanism is introduced to automatically trigger an alarm to notify relevant personnel to intervene when an abnormality occurs in the processing pipeline or performance indicators exceed the threshold.

[0048] Furthermore, the S4 includes:

[0049] S41. On the Kafka consumer side, receive Nginx access log data from the Kafka message queue; parse each log data into structured information;

[0050] S42. Parse the Request-URL in each log and extract the key parts, including the path and query parameters.

[0051] S43, removing irrelevant parts from the URL and standardizing the URL into a comparable format;

[0052] S44. Further classify the standardized URL into different business categories according to preset business rules or regular expressions; generate an aggregation key based on the classified business categories and the standardized URL path.

[0053] S45. For each aggregation key, count the number of corresponding log entries as the basic data of the monitoring indicator; and further count the number of occurrences of different HTTP status codes under each aggregation key;

[0054] S46. Set a time window and aggregate the aggregation key within each time window to generate time series data. Dynamically adjust the aggregation granularity based on the number of monitoring indicators and system load.

[0055] S47. Regularly clean up aggregated data that has not been updated for a long time, convert the aggregated data into monitoring indicators, and output the monitoring indicators to the monitoring system or storage system.

[0056] Furthermore, the S5 includes:

[0057] S51. Configure the rule engine, define rules for identifying abnormal logs, and integrate machine learning models to automatically discover potential performance issues or abnormal patterns.

[0058] S52. Calculate key performance indicators in real time, including request response time, access volume, and error rate, and update them to the monitoring system;

[0059] S53. Key performance indicators are displayed through the monitoring interface, and based on the alarm mechanism, when KPIs exceed the preset threshold, an alarm is automatically triggered to notify relevant personnel.

[0060] Furthermore, the S6 includes:

[0061] S61. Dynamically generate real-time monitoring indicators based on business needs, customize monitoring items and thresholds, and update monitoring data in real time;

[0062] S62. Set the validity period of monitoring indicators, automatically clean up expired and unupdated indicators, and perform differentiated optimization of the cleanup strategy based on data access frequency;

[0063] S63. Monitor the performance of the entire log monitoring system and optimize the system based on the monitoring results.

[0064] The beneficial effects of the present invention are: Filebeat captures Nginx logs in real time and quickly pushes them to Kafka, ensuring that the monitoring system can respond to changes in the logs in a timely manner; Kafka's high throughput characteristics can be used to efficiently process large amounts of log data, while reducing the load of network transmission through compression algorithms; through intelligent aggregation algorithms, request URLs with the same business meaning are summarized into monitoring indicator sets, simplifying the complexity of monitoring data and improving the efficiency of data analysis; the built-in rule engine and machine learning algorithm can deeply analyze log data, extract key performance indicators, and help timely discover and solve potential performance problems; according to business needs and system configuration , generate and display various real-time monitoring indicators, making the monitoring results more intuitive and easy to understand; regular monitoring indicator expiration processing mechanism ensures the data accuracy and system performance of the monitoring system; in the process of pushing log data, through the error retry mechanism and idempotence or transaction characteristics, ensure the integrity and consistency of the data; the solution supports the definition of multiple processing pipelines according to business needs and technical architecture, which improves the scalability and flexibility of the system; through performance monitoring and tuning of the entire log monitoring system, the system performance can be continuously optimized and the processing efficiency can be improved; the introduction of an alarm mechanism can promptly notify relevant personnel for processing when the monitoring indicators exceed the preset threshold or the processing pipeline has an abnormality. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a step diagram of the solution described in the present invention. DETAILED DESCRIPTION

[0066] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0067] One embodiment of the present invention, as Figure 1 As shown, an nginx monitoring indicator solution for a message queue includes:

[0068] S1. Deploy Filebeat as a log collector on the Nginx server to capture Nginx access logs and error logs in real time.

[0069] S2 and Filebeat encapsulate the collected log data into messages and push them to the Kafka message queue through the TCP protocol;

[0070] S3, consumers (such as custom Kafka consumer services) pull log messages from Kafka and parse and process the Nginx access logs in the messages;

[0071] S4. Aggregate Request-URLs with the same business meaning through an intelligent aggregation algorithm and summarize them into a limited set of monitoring indicators;

[0072] S5. Through the built-in rule engine and machine learning algorithm, consumers further analyze the log data, extract key performance indicators (such as request response time, access volume, error rate, etc.), and update them to the monitoring system in real time;

[0073] S6. Generate various real-time monitoring indicators based on business needs and system configuration, and display them in the monitoring system; and implement a regular monitoring indicator expiration processing mechanism to automatically clean up monitoring indicators that have not been updated within a certain period of time (such as N hours).

[0074] The working principle of the above technical solution is as follows: Filebeat, as a lightweight log collection tool, is deployed on the Nginx server. It can monitor and collect access logs (such as access.log) and error logs (such as error.log) generated by Nginx in real time; Filebeat captures new log entries in real time by reading the contents of the log files. Once a new log entry is detected, Filebeat will process it immediately; Filebeat will encapsulate the captured log entries into a message format for subsequent transmission and processing. Each message usually contains key information such as the log timestamp, log level, log content, etc.; the encapsulated log message is pushed to the Kafka message queue via TCP. As a high-performance message middleware, Kafka can ensure high availability and scalability of messages; Kafka's consumer service (which can be a custom or existing log processing service) regularly pulls log messages from Kafka. These messages are organized according to Kafka's partitioning and offset mechanisms. The consumer service parses the pulled log messages, particularly Nginx access logs, extracting key information such as the request method, URL, response time, and status code. Using a specific intelligent aggregation algorithm, request-URLs with the same business significance are aggregated. This means similar requests can be treated as the same monitoring object, simplifying monitoring logic and reducing data redundancy. The aggregated data is summarized into a limited set of monitoring metrics covering key business areas such as request volume, response time distribution, and error rates. A built-in rules engine and machine learning algorithms are used to conduct in-depth analysis of log data. The rules engine identifies potential anomalies or violations, while the machine learning algorithms discover hidden patterns or trends in the data. Key performance indicators (KPIs) such as request response time, number of visits, and error rates are extracted from the log data and updated in real time to the monitoring system. Based on business needs and system configuration, various real-time monitoring metrics are generated and displayed in the monitoring system. These indicators are presented in charts and dashboards, helping operations personnel quickly understand system status. To prevent monitoring data from becoming outdated or taking up too much storage space, the system implements a regular monitoring indicator expiration mechanism. Monitoring indicators that have not been updated within a certain period of time (e.g., N hours) are automatically cleared to maintain the timeliness and accuracy of monitoring data.

[0075] The effects of the above technical solution are as follows: Filebeat can capture the access logs and error logs of Nginx in real time, ensuring the freshness and timeliness of data; through the Kafka message queue, log data is pushed to the processing system in an efficient manner, reducing the risk of data transmission delay and loss; the Kafka consumer service can quickly pull and process log messages from the queue, achieving fast parsing and aggregation of log data; components such as Filebeat and Kafka can be flexibly deployed on the Nginx servers that need to be monitored, or deployed as an independent log processing cluster; Kafka supports horizontal scaling, and as the amount of logs increases, more Kafka nodes can be easily added to process more log data; the consumer service can be customized according to specific requirements to flexibly process log data and extract the required monitoring metrics; through intelligent aggregation algorithms, Request-URLs with the same business meaning are aggregated, simplifying the monitoring logic and reducing data redundancy; the built-in rule engine and machine learning algorithms can deeply analyze log data, discover potential problems and trends, and improve the accuracy and intelligence level of monitoring; the monitoring system can display various monitoring metrics in real time to help operation and maintenance personnel quickly understand the system status; when the monitoring metrics exceed the preset thresholds, the system can automatically trigger alarms and notify relevant personnel for processing in a timely manner; a regular monitoring metric expiration processing mechanism is executed to clean up outdated data and maintain the timeliness and accuracy of monitoring data; through data aggregation and intelligent analysis, the amount of data that needs to be stored is reduced, optimizing the use of storage resources; this solution can be easily integrated with existing monitoring systems, alarm systems, etc. to achieve unified management and monitoring of log data; as the business develops, the monitoring scope can be easily expanded, monitoring metrics can be increased, or the processing logic can be optimized to meet the constantly changing monitoring requirements.

[0076] In one embodiment of the present invention, S1 includes:

[0077] S11. Configure the Nginx server, including the log file path, format (such as combinedlogformat), and permission settings. Install Filebeat on the Nginx server, configure Filebeat, and monitor the access log (access.log) and error log (error.log) of Nginx through Filebeat;

[0078] S12. Configure the input module of Filebeat, specify the log file path, and set the buffer and backoff strategies;

[0079] S13. Configure the output module of Filebeat and send log data to Kafka, including the Kafka cluster address, authentication information, etc.

[0080] S14. Start the Filebeat service to capture new entries in the Nginx log file in real time and perform preliminary data cleaning in Filebeat, such as removing blank lines and filtering invalid logs.

[0081] S15. Mark the log according to its type (access or error) and add metadata, where the metadata includes a timestamp and a server identifier.

[0082] S16. Parse the log entry using a regular expression or a JSON parser (if the log format is JSON) to extract key information, including the IP address, request URL, and status code.

[0083] The working principle of the above technical solution is as follows: configure the Nginx server to ensure that the path, format (such as combinedlogformat) and permissions of the log files (such as access.log and error.log) are set correctly. These configurations determine how Nginx generates and stores logs, and what information will be recorded; install Filebeat on the Nginx server, which is a key component for log collection. After the installation is complete, make the necessary configurations to specify how Filebeat interacts with Nginx; configure the input module of Filebeat and specify the path of Nginx's access log and error log files. In this way, Filebeat will know where to read the log data; set appropriate buffer and backoff strategies to optimize the log reading process. Buffer is used to temporarily store log data before sending it to the output end, and the backoff strategy defines how Filebeat retries to send data when the network fails or the output end is temporarily unavailable; configure the output module of Filebeat, specify the address of the Kafka cluster and the necessary authentication information (if the Kafka cluster has security authentication enabled). This allows Filebeat to know which Kafka topic to send data to. After starting the Filebeat service, it monitors new entries in the Nginx log files in real time. Whenever new log entries are generated, Filebeat immediately captures them. While capturing log entries, Filebeat performs preliminary data cleaning operations, such as removing blank lines and filtering invalid logs, to reduce unnecessary data transmission and processing burden. Based on the log entry's content (such as the log file name or specific tags in the log format), Filebeat tags the captured log entries as either access log or error log. Each log entry is given metadata, such as a timestamp (recording the time the log entry was generated) and a server identifier (identifying the Nginx server that generated the log). This metadata facilitates subsequent log processing and analysis. Depending on the Nginx log format (such as combinedlogformat or JSON), Filebeat parses the log entries using regular expressions or a JSON parser. This process extracts key information from the log entries, such as the IP address, request URL, and status code. This extracted information is used for subsequent log processing, aggregation, analysis, and monitoring. For example, IP addresses and request URLs can be used to identify different user requests and business operations; status codes can be used to evaluate the success rate or error rate of requests.

[0084] The effect of the above technical solution is: Filebeat can capture new entries in Nginx's access log and error log in real time to ensure the timeliness and accuracy of the data. This is crucial for timely discovery and response to system problems; by optimizing the buffer and backoff strategies, Filebeat can efficiently process log data and send it to the Kafka message queue, reducing the risk of data transmission delays and losses; Filebeat's input and output modules support flexible configuration options, which can be customized according to the specific format of the Nginx log and the configuration of the Kafka cluster; as the amount of Nginx logs increases, the capacity of the Kafka cluster can be easily expanded to process more log data. At the same time, Filebeat itself also supports multi-instance deployment to further improve the efficiency of log collection; before sending log data to Kafka, Filebeat will perform preliminary data cleaning, such as removing blank lines, filtering invalid logs, etc. This helps reduce data noise in subsequent processing and improves data quality. Filebeat adds metadata such as timestamps and server identifiers to each log entry, which are very important for subsequent data analysis and monitoring. Through regular expressions or JSON parsers (depending on the log format), Filebeat can accurately parse log entries and extract key information such as IP addresses, request URLs, and status codes. This information is crucial for understanding user behavior, monitoring system performance, and locating problems. The extracted key information provides a rich data source for subsequent log analysis, allowing operations and maintenance personnel to gain a deeper understanding of system status and user behavior. Sending Nginx log data to Kafka in real time provides a solid foundation for subsequent monitoring and alarm systems. Operations and maintenance personnel can build real-time monitoring indicators and alarm rules based on this data to promptly detect and respond to system problems. Since data is captured and transmitted in real time, when a system problem occurs, an alarm can be triggered quickly, helping operations and maintenance personnel to quickly locate and resolve the problem. Filebeat and Kafka are both widely used open source tools that support multiple log formats and message queue protocols, so they can be easily integrated with existing monitoring and alarm systems. As the business develops, the monitoring scope of Filebeat or the cluster size of Kafka can be easily expanded to meet changing monitoring needs.

[0085] In one embodiment of the present invention, the S2 includes:

[0086] S21. Encapsulate the parsed log data into a Kafka message and compress the Kafka message using a compression algorithm;

[0087] S22. Configure Filebeat to push the encapsulated message to the Kafka cluster using the TCP protocol.

[0088] During the push process, the error retry mechanism ensures that log data is safely resent when the network fluctuates or the Kafka cluster is temporarily unavailable.

[0089] S24: Monitor the push process and record key indicators, including push success rate and delay.

[0090] The above technical solution works as follows: Log data parsed by Filebeat (including complete log information and metadata such as timestamps, server identifiers, IP addresses, request URLs, and status codes) is encapsulated into Kafka messages. Each Kafka message should contain sufficient information to restore the original log entry during subsequent processing. To reduce network transmission burden and improve efficiency, Kafka messages are compressed using compression algorithms (such as Gzip and Snappy). Compressed data consumes less bandwidth during transmission, resulting in faster transmission and lower latency. Filebeat is configured to push encapsulated messages to the Kafka cluster using TCP. The native Kafka protocol generally offers lower latency and higher throughput, making it preferred when possible. Filebeat interacts with the Kafka cluster through its built-in Kafka output plugin, pushing messages to the specified Kafka topic. The Kafka cluster is responsible for receiving, storing, and distributing these messages. During the push process, if there are abnormalities such as network fluctuations or temporary unavailability of the Kafka cluster, Filebeat triggers an error retry mechanism. This means that it will try to resend failed log data until it succeeds or reaches the preset retry limit. This mechanism ensures the security and integrity of the log data; during the retry process, Filebeat will ensure that the log data that has been successfully pushed will not be sent repeatedly. This is usually achieved by tracking the status of sent messages or leveraging the idempotence and transactional features of Kafka; the process of Filebeat pushing log data to the Kafka cluster is monitored in real time. This includes monitoring the network status, the health of the Kafka cluster, and the performance indicators of Filebeat; recording and storing key indicators in the push process, such as the push success rate (the ratio of the number of successfully pushed messages to the total number of messages) and latency (the time difference from when Filebeat receives the log data to when it is successfully pushed to the Kafka cluster). These indicators are critical for evaluating push performance, diagnosing problems, and optimizing system configuration.

[0091] The above technical solution achieves the following benefits: By compressing Kafka messages, the amount of data transmitted over the network can be significantly reduced, thereby reducing network bandwidth consumption and transmission costs. This is particularly important for transmitting large amounts of log data. Compressed data requires less time to transmit, which speeds up log data push and improves overall system processing efficiency. The TCP protocol can be selected for message push based on actual conditions. This flexibility allows the solution to adapt to different network environments and Kafka cluster configurations. Using the native Kafka protocol generally achieves lower latency and higher throughput, further improving system performance. In the event of network fluctuations or temporary Kafka cluster unavailability, an error retry mechanism ensures the secure retransmission of log data. This mitigates the risk of data loss and ensures reliable data transmission. During the retry process, data consistency and integrity are ensured by avoiding the retransmission of successfully pushed log data. Real-time monitoring of the push process allows for timely detection and response to potential issues, such as network failures and Kafka cluster anomalies. This helps operations personnel quickly identify and take appropriate measures. Key metrics, such as push success rate and latency, are recorded and stored, providing important insights for performance tuning. By analyzing these metrics, we can identify system bottlenecks and optimize related configurations, further improving system performance. Each sub-step (such as data encapsulation, compression, push, and monitoring) is relatively independent and modular, making the system easy to expand and maintain. When new features need to be added or existing features optimized, the corresponding modules can be easily modified or replaced. Using Kafka as the storage and distribution platform for log data allows the system to seamlessly integrate with other Kafka-based applications or tools, reducing system integration costs and complexity.

[0092] In one embodiment of the present invention, the step S21 includes:

[0093] Extract key fields from the log data parsed by Filebeat, including timestamp, log level, client IP, request method, Request-URL, status code, request size, and response time;

[0094] According to the Kafka message format requirements, the key fields are structured and stored as key-value pairs or JSON format, and necessary metadata is added to each structured log data; such as the log source (Nginx server identifier), log type (access log or error log), processing timestamp, etc., and the integrity and accuracy of the metadata are verified;

[0095] Use the Kafka client library (such as KafkaProducer) to create a Kafka message object; encapsulate the structured log data and its metadata into the Kafka message body, set the message topic (Topic) and partition key (PartitionKey); and configure the message attributes, such as the message priority and persistence strategy;

[0096] Compress the Kafka message body using a compression algorithm (such as GZIP, Snappy, LZ4, etc.) and evaluate the compression effect.

[0097] The above technical solution works as follows: It obtains raw log data from Filebeat parsed log data; extracts key fields such as timestamp, log level, client IP address, request method, request URL, status code, request size, and response time based on the log data format (such as Nginx's combinedlogformat or a custom format); stores key fields in either key-value pairs or JSON format, depending on the Kafka message format. The key-value pair format is simple and straightforward, suitable for simple data models; while the JSON format is more flexible and can easily represent complex data structures; adds necessary metadata to each structured log data entry, such as the log source (Nginx server identifier), log type (access log or error log), and processing timestamp. This metadata helps quickly locate and identify log entries when storing and querying log data in Kafka; verifies the integrity and accuracy of the added metadata to ensure it can be correctly identified and used during message transmission and subsequent processing; creates a Kafka message object using a Kafka client library (such as KafkaProducer); and encapsulates the structured log data and its metadata into the Kafka message body. This includes setting the topic of the message, which determines which partition in the Kafka cluster the message will be sent to; and setting the partition key, which is used to determine the storage location of the message within the partition; configuring other message attributes as needed, such as the priority of the message (although Kafka itself does not directly support priority, it can be achieved through other means, such as using different topics or consumer groups to process messages of different priorities), persistence strategy (such as setting the replication factor and synchronization strategy of the message), etc.; selecting an appropriate compression algorithm (such as GZIP, Snappy, LZ4, etc.) to compress the Kafka message body according to actual needs. These algorithms have their own advantages and disadvantages in compression rate and compression speed, and need to be selected according to actual conditions; evaluating the compressed message body to ensure that while reducing the data volume, it will not significantly affect the performance and efficiency of subsequent processing. This usually involves testing and analyzing metrics such as compression ratio, decompression speed, and CPU usage; sending the compressed Kafka message to the specified topic and partition in the Kafka cluster; after receiving the message, the Kafka cluster will store and distribute it according to the configuration so that subsequent consumers (such as log analysis systems, monitoring systems, etc.) can consume and process it.

[0098] The above technical solution makes log data more standardized and easier to understand by extracting key fields and storing them in a structured key-value pair or JSON format. This helps reduce the complexity of data parsing and improves data processing efficiency. Structured log data makes it easier to store and query in Kafka. Whether using stream processing frameworks such as KafkaStreams and SparkStreaming, or exporting data to other systems such as Elasticsearch and Hadoop through connectors such as KafkaConnect, log data can be queried and analyzed more efficiently. Essential metadata (such as log source, log type, and processing timestamp) is added to each structured log data entry, enhancing data traceability. This allows for quick identification of specific log entries and the associated Nginx server when problems arise, enabling troubleshooting and remediation. Verifying the integrity and accuracy of metadata ensures the correctness and reliability of data during transmission and subsequent processing. This helps reduce analytical bias and erroneous decisions caused by data errors or inconsistencies. Compression of Kafka messages using compression algorithms significantly reduces the data size, thereby alleviating the burden on network transmission. This facilitates fast log data transmission in environments with limited bandwidth or high network latency. The compressed data also improves storage efficiency within the Kafka cluster. The reduced data size allows for more log data to be stored without increasing storage space. When creating Kafka message objects using the Kafka client library, you can flexibly configure message properties (such as message priority and persistence policy). This allows you to tailor message processing and storage strategies to meet diverse business scenarios and performance requirements. By configuring the topic and partition key, you can achieve partitioned storage and parallel processing of log data. This improves system scalability and processing performance, especially when processing large amounts of log data. Compressed data is evaluated to ensure that the reduction in size does not significantly impact the performance and efficiency of subsequent processing. This helps select the most appropriate compression algorithm and configuration parameters to achieve the optimal balance between compression and performance. By monitoring and recording key metrics (such as compression ratio, decompression speed, and CPU usage), you can continuously optimize system performance and stability. This helps you identify and resolve potential issues promptly, ensuring long-term stable system operation.

[0099] In one embodiment of the present invention, the step S23 includes:

[0100] When pushing log data to the Kafka cluster, the status code and response information of the push operation are monitored in real time. Errors are captured and identified using the exception handling mechanism provided by the Kafka client library. These error conditions include network anomalies and Kafka cluster unavailability.

[0101] Classify errors to determine whether they are retryable errors (such as network fluctuations and temporary service unavailability) or non-retryable errors (such as data format errors and long-term Kafka cluster unavailability);

[0102] If it is a retryable error, a retry is performed based on a preset retry strategy, which includes the number of retries, the retry interval, and data integrity verification during retries; and based on the exponential backoff strategy, a waiting time is added between each retry;

[0103] Before retrying, the integrity and consistency of the log data to be resent are checked. Through Kafka's idempotence or transactions, duplicate or inconsistent data can be avoided after network partition or Kafka cluster failure recovery.

[0104] During the resending process, the detailed information of each retry is recorded, including the retry time, number of retries, and the size of the resent data. For successfully resent log data, the relevant status information is updated and the relevant systems or personnel are notified.

[0105] If the maximum number of retries or the total retry time limit is reached and the attempt is still unsuccessful, the failed log data will be recorded in the error log and an alarm will be triggered to notify relevant personnel to intervene;

[0106] Analyze the reasons for retry failures and adjust the retry strategy or optimize the system configuration based on the analysis results.

[0107] The above technical solution works as follows: During the process of pushing log data to the Kafka cluster, the exception handling mechanism provided by the Kafka client library is utilized to monitor the status code and response information of the push operation in real time. Once an error condition (such as network anomalies or Kafka cluster unavailability) is detected, the exception is immediately captured and identified. Captured errors are classified into retryable and non-retryable errors. Retryable errors typically include network fluctuations and brief service unavailability. These errors may be temporary and can be successfully resolved through retrying. Non-retryable errors involve more serious issues, such as data format errors or long-term Kafka cluster unavailability, which require manual intervention. For retryable errors, retries are performed based on a preset retry strategy. The retry strategy includes the number of retries, the retry interval, and data integrity verification during retries. An exponential backoff strategy is used to increase the waiting time between each retry to reduce the impact on the Kafka cluster. Furthermore, a maximum number of retries and a total retry time limit are set to avoid indefinite retries that waste resources. Before retrying, perform integrity and consistency checks on the log data to be resent to ensure that the data has not been tampered with or corrupted. Leverage Kafka's idempotence and transactions to ensure that resent data is not duplicated or inconsistent after network partitions or Kafka cluster failure recovery. During the resend process, detailed information about each retry is recorded, including the retry time, number of retries, and the size of the resent data. For successfully resent log data, relevant status information is updated and relevant systems or personnel are notified. If the maximum number of retries or the total retry time limit is reached and the resend fails, the failed log data is recorded in the error log. An alarm is triggered to notify relevant personnel to intervene and further analyze the cause of the problem and take remedial measures. Conduct in-depth analysis of the reasons for the retry failure to identify the root cause of the error. Based on the analysis results, adjust the retry strategy or optimize the system configuration to reduce the likelihood of similar errors in the future.

[0108] The above technical solution achieves the following benefits: real-time monitoring of the status code and response information of push operations enables timely detection and capture of errors, enabling rapid response and resolution. This helps reduce system instability or data loss caused by undetected errors. Errors are categorized as retryable and non-retryable, with different error handling strategies implemented for each type. For retryable errors, multiple attempts are made using a pre-set retry strategy, improving the system's fault tolerance and stability. Setting a maximum number of retries and a total retry time limit prevents resource waste caused by indefinite retries. This helps allocate system resources efficiently and ensures that critical services are not impacted. An exponential backoff strategy increases the waiting time between retries, reducing frequent impacts on the Kafka cluster while increasing the success rate of retries. Before retrying, integrity and consistency checks are performed on the log data to be retransmitted to ensure that the data has not been tampered with or corrupted. This helps maintain data accuracy and reliability. Leveraging Kafka's idempotence and transactional properties, duplicate or inconsistent data is avoided in retransmitted data after a network partition or Kafka cluster failure recovery. This further enhances data integrity and consistency. During the retransmission process, detailed information about each retry is recorded, including the retry time, number of retries, and the size of the retransmitted data. This information facilitates subsequent problem diagnosis and analysis. If a failure is detected after reaching the maximum number of retries or the total retry time limit, the failed log data is recorded in the error log, and an alarm is triggered to notify relevant personnel to intervene. This helps identify problems promptly and take appropriate remedial measures. The cause of the retry failure is analyzed in depth to identify the root cause of the error. This helps identify potential problems in the system and take appropriate optimization measures. Based on the analysis results, the retry strategy is adjusted or the system configuration is optimized to reduce the likelihood of similar errors in the future. This continuous improvement mechanism helps improve the overall performance and stability of the system.

[0109] In one embodiment of the present invention, S3 includes:

[0110] S31. Deploy the Kafka consumer service, configure the consumer group, subscription topic (corresponding to Nginx logs), and partition allocation strategy, and perform automatic consumer offset management.

[0111] S32: The consumer pulls log messages from Kafka, decompresses them, parses each message, and recovers the original Nginx log data and its metadata.

[0112] S33. Distribute the parsed data to different processing pipelines based on the log type (access or error).

[0113] The above technical solution works as follows: Deploy the Kafka consumer service on the appropriate server. This typically involves installing the Kafka client library (such as the Java Kafka client library) and necessary dependencies; configure a consumer group for each consumer service instance. A consumer group is a concept in Kafka used to achieve message load balancing and high availability. Consumers within the same consumer group share the load of subscribed topics; configure consumers to subscribe to specific Kafka topics, which correspond to Nginx logs. At the same time, set a partition distribution strategy, such as Round Robin or Range distribution, to ensure that messages are evenly distributed to consumers within the consumer group; configure the consumer to automatically manage its offset in Kafka. The offset records the position of the message that the consumer has read. Automatic offset management ensures that after the consumer restarts or fails, it can continue reading messages from the last stopped position, thus avoiding message duplication or loss; the consumer service periodically pulls log messages from the subscribed topics from Kafka. These messages may have been compressed before being sent to improve transmission efficiency, so consumers need to perform decompression operations first; parse each pulled message to restore the original Nginx log data and its metadata. This usually involves decoding the message format (such as JSON, Protobuf, etc.) and extracting key information in the log, such as request time, request URL, status code, etc.; after parsing the log data, classify it according to the log type (access log or error log). This is usually done by checking specific fields or formats in the log; distribute the classified log data to different processing pipelines. The processing pipeline can be different data processing services, analysis systems, or storage systems, depending on business needs. For example, access logs may be sent to a real-time analysis system to monitor website traffic, while error logs may be sent to a log management system for troubleshooting.

[0114] The above technical solution ensures that messages are consumed in the order they were generated in Kafka by configuring consumer groups and automatic offset management, which is particularly important for applications that require preserving event order. Automatic offset management also prevents message loss due to consumer failures or restarts, as consumers can continue consuming messages from where they left off. Using Kafka as a message queue leverages Kafka's high availability and fault tolerance. Even if some nodes in the Kafka cluster fail, consumers can still pull messages from other available nodes, ensuring stable system operation. Consumers within a consumer group can collaborate and share the message processing load, improving overall system processing capacity and reliability. Using a partition allocation strategy, Kafka distributes messages across different partitions, allowing consumers within a consumer group to process messages from these partitions in parallel, thereby increasing message processing throughput. Consumers can dynamically adjust the rate at which they pull messages based on their processing capacity, avoiding performance bottlenecks caused by overutilized resources. Consumers can decompress and parse the pulled log messages to recover the original Nginx log data and metadata. This enables consumers to further process and analyze log data as needed. The parsed data is distributed to different processing pipelines based on the log type (access or error), enabling flexible routing and processing of data and meeting the diverse data processing requirements of different business needs. Kafka's high throughput and low latency enable the S3 technical solution to support real-time log processing. Consumers can pull log messages from Kafka in almost real time and process and analyze them, which is especially important for application scenarios that require rapid response (such as real-time monitoring, fault warning, etc.). The modular design allows each component to be expanded and maintained independently. For example, when more consumers need to be added to process log messages, consumer instances can be easily added and partition allocation strategies can be adjusted. Kafka's cluster management function also simplifies the system's operation and maintenance, allowing administrators to more easily monitor and manage the status and performance of the Kafka cluster.

[0115] In one embodiment of the present invention, the step S33 includes:

[0116] When the consumer pulls and parses log messages from Kafka, it obtains specific fields in the metadata or content of each message (such as log level, log type tag, etc.) and identifies the log type (access log or error log, and creates a log type enumeration or dictionary;

[0117] Based on business needs and technical architecture, define multiple processing pipelines, each responsible for processing a specific type of log data. For example, an access log processing pipeline may include steps such as data analysis and report generation; an error log processing pipeline may include steps such as anomaly detection and alarm notification.

[0118] Configure the input, output, processing logic, and required resources (such as computing resources, storage resources, etc.) of each processing pipeline;

[0119] Write data distribution logic to route parsed data to the corresponding processing pipeline based on log type. Use load balancing to evenly distribute data to various processing pipelines in high-concurrency situations to avoid overloading a single pipeline. Process data asynchronously.

[0120] Establish a communication mechanism between pipelines to share and communicate data between pipelines through message queues, event buses, or distributed caches;

[0121] After completing data processing, each processing pipeline generates monitoring indicators in the format of the monitoring platform and reports them to the monitoring platform.

[0122] An alarm mechanism is introduced to automatically trigger an alarm to notify relevant personnel to intervene when an abnormality occurs in the processing pipeline or performance indicators exceed the threshold.

[0123] The above technical solution works as follows: the consumer pulls log messages from Kafka; parses the metadata or content of each message to obtain specific fields (such as log level and log type tag); identifies the log type (access log or error log) and creates a log type enumeration or dictionary to quickly match and identify log types, improving processing efficiency; Based on business needs and technical architecture, multiple processing pipelines are defined; each pipeline is responsible for processing a specific type of log data. For example, an access log processing pipeline may include steps such as data analysis and report generation; an error log processing pipeline may include steps such as anomaly detection and alert notification. Configure the input, output, processing logic and required resources of each processing pipeline; write data distribution logic to route the parsed data to the corresponding processing pipeline according to the log type; implement data distribution through conditional judgment statements, message routing middleware or stream processing framework; introduce a load balancing mechanism to evenly distribute data to each processing pipeline in high concurrency situations to avoid overloading a single pipeline; process log data asynchronously to ensure that the distribution and processing of log data will not block the message pulling process of Kafka consumers, thereby improving overall processing efficiency; establish a communication mechanism between pipelines to share and communicate data between pipelines through message queues, event buses or distributed caches; after each processing pipeline completes data processing, the processing results are transmitted to the next layer system for use; feedback on results is achieved through logging, database updates, API calls, etc.; real-time monitoring of the performance indicators of the processing pipeline, such as processing speed, latency, error rate, etc.; introduce an alarm mechanism to automatically trigger an alarm to notify relevant personnel to intervene when an abnormality occurs in the processing pipeline or the performance indicators exceed the threshold.

[0124] The above technical solution achieves the following benefits: By quickly matching and identifying log types (using log type enumeration or dictionary), data processing time is reduced and processing efficiency is improved. The asynchronous processing mechanism ensures that log data distribution and processing does not block Kafka consumers' message pulling process, further improving the system's overall processing capacity and responsiveness. Multiple processing pipelines are defined, each responsible for processing a specific type of log data. This modular design makes the system easily scalable. When new log types or processing steps need to be processed, new processing pipelines can be easily added or the configuration of existing pipelines modified. The load balancing mechanism ensures that processing tasks are evenly distributed in high-concurrency scenarios, avoiding overloading a single pipeline and thus improving the system's scalability and stability. Detailed configuration of each processing pipeline's input, output, processing logic, and required resources helps optimize resource utilization. The system can dynamically adjust resource allocation based on actual needs, avoiding resource waste and bottlenecks. Inter-pipeline communication mechanisms (such as message queues and event buses) allow the system to efficiently share and transfer data between different processing stages, further improving resource utilization efficiency. By defining different processing pipelines and configuring specific processing logic for each pipeline, the system can flexibly process log data according to different business needs. This flexibility helps meet the data analysis needs of different departments and teams. Accurate log type identification and processing logic ensure the accuracy of data processing and reduce the possibility of errors and misunderstandings; real-time monitoring of the performance indicators of the processing pipeline (such as processing speed, latency, error rate, etc.) helps to promptly identify and resolve potential problems and ensure the stable operation of the system; introducing an alarm mechanism to automatically trigger alarm notifications when anomalies occur in the processing pipeline or performance indicators exceed thresholds can quickly attract the attention of relevant personnel and speed up problem response and resolution. An efficient log processing system can generate valuable analysis reports and alarm information more quickly, helping business personnel better understand the system operation status and business trends, thereby improving user experience and satisfaction. By promptly processing error logs and triggering alarm notifications, the system can resume normal operation more quickly, reduce the impact of system failures on the business, and enhance user trust.

[0125] In one embodiment of the present invention, the S4 includes:

[0126] S41. On the Kafka consumer side, receive Nginx access log data from the Kafka message queue; parse each log data into structured information, including but not limited to timestamp, client IP, request scheme, Request-URL, status code, request size, response time, etc.

[0127] S42. Parse the Request-URL in each log and extract key parts, including the path and query string.

[0128] S43. Remove irrelevant parts of the URL (such as dynamic parameters, timestamps, etc.) and standardize the URL into a comparable format; for example, standardize http: / / example.com / product?id=123&date=2023-04-01 to http: / / example.com / product.

[0129] S44. Further classify the standardized URLs into different business categories according to preset business rules or regular expressions; generate an aggregation key (AggregationKey) based on the classified business categories and the standardized URL paths;

[0130] S45. For each aggregation key, count the number of corresponding log entries as the basic data of the monitoring indicator; and further count the number of occurrences of different HTTP status codes under each aggregation key;

[0131] S46. Set a time window (e.g., 1 minute, 5 minutes, etc.), aggregate the aggregation key within each time window, and generate time series data; dynamically adjust the aggregation granularity based on the number of monitoring indicators and the system load;

[0132] S47. Regularly clean up aggregated data that has not been updated for a long time to avoid unlimited growth in the number of monitoring indicators. Convert the aggregated data into monitoring indicators, such as request volume, average response time, error rate, etc., and output the monitoring indicators to the monitoring system or storage system.

[0133] The above technical solution works as follows: The processing program deployed on the Kafka consumer side is responsible for pulling Nginx access log data from the Kafka message queue in real time. It parses each log data entry from its original format (e.g., text) into structured information. This information includes, but is not limited to, timestamp, client IP address, request scheme (e.g., HTTP / HTTPS), request URL, status code, request size, response time, etc. This step is the foundation for subsequent processing, enabling the program to understand and process the log data in a structured manner. The request URL in each log entry is deeply parsed to extract key components, such as the path and query string. These components are crucial for understanding the context and purpose of the request. To ensure URL comparability, irrelevant components, such as dynamic parameters (e.g., user ID, session ID) and timestamps, are removed. These components are often noise for monitoring and aggregated analysis; removing them helps more accurately reflect the business meaning of the request. The processed URL is converted into a standardized format for subsequent classification and aggregation operations. The standardized URLs are further classified into different business categories based on pre-defined business rules or regular expressions. This step is to group requests with similar business implications into the same category to facilitate subsequent monitoring and analysis. Based on the classified business categories and standardized URL paths, an aggregation key (AggregationKey) is generated. The aggregation key is the basis for subsequent aggregation operations, ensuring that request-URLs with the same business implications are grouped under the same monitoring indicator. For each aggregation key, the corresponding log entries are counted as the basic data for the monitoring indicator. The number of occurrences of different HTTP status codes under each aggregation key is further counted to analyze key performance indicators such as the success rate and error rate of the request. A time window (such as 1 minute, 5 minutes, etc.) is set to aggregate the aggregation key within each time window to generate time series data. The aggregation granularity is dynamically adjusted based on the number of monitoring indicators and the system load. This helps balance the level of monitoring detail and system performance, ensuring sufficient monitoring data without excessively consuming system resources. Aggregated data that has not been updated for a long time is regularly cleaned up to prevent the unlimited growth of the number of monitoring indicators. This step helps keep the monitoring system clean and efficient. Aggregated data is converted into monitoring metrics, such as request volume, average response time, and error rate, and these metrics are exported to the monitoring system or storage system. This allows business personnel and system administrators to understand the system's operating status in real time and identify and resolve problems promptly.

[0134] The above technical solution achieves this by receiving Nginx access log data in real time through Kafka consumers, ensuring timely and real-time data processing. This is crucial for systems requiring rapid response and real-time monitoring. URLs are parsed and standardized, removing irrelevant portions and making different URLs comparable. This helps more accurately analyze user request patterns and trends, identifying potential issues or anomalies. Standardized URLs are categorized into different business categories based on pre-set business rules or regular expressions, and aggregation keys are generated. This flexible classification and aggregation mechanism enables the system to provide customized monitoring and analysis based on diverse business needs. The generation of aggregation keys ensures that request URLs with the same business meaning are grouped under the same monitoring metric, simplifying the number of monitoring metrics and improving monitoring efficiency and accuracy. Statistics are collected for the number of log entries and the occurrence of different HTTP status codes under each aggregation key, enabling in-depth analysis of key performance indicators such as request success rates and error rates. This is crucial for optimizing system performance and improving user experience. Dynamically adjusting the aggregation granularity based on the number of monitoring metrics and system load balances monitoring detail and system performance. This adaptive adjustment mechanism enables the system to maintain optimal operating status in different scenarios. Regularly cleaning aggregated data that has not been updated for a long time prevents the uncontrolled growth of monitoring indicators, saving storage and computing resources. Aggregated data is converted into monitoring indicators and output to the monitoring system or storage system, allowing business personnel and administrators to easily view and analyze monitoring data. By real-time monitoring and in-depth analysis of Nginx access log data, the system can promptly identify and address potential problems or anomalies, improving system stability and reliability. Flexible classification and aggregation mechanisms, as well as configurable monitoring indicators, make the system easier to maintain and expand, reducing maintenance costs.

[0135] In one embodiment of the present invention, the step S46 includes:

[0136] Based on business needs and log generation rate, set the initial time window size (e.g., every 5 minutes, every hour, etc.); introduce an adaptive algorithm to dynamically adjust the time window size based on the real-time change rate of monitoring data (e.g., a surge or drop in log volume); and set an overlap strategy between time windows (e.g., 50% overlap) to prevent data omissions.

[0137] Within the time window, log data is grouped according to the generated aggregation key, and the number of log entries in each group is counted as basic monitoring data;

[0138] Count the number of occurrences of different HTTP status codes (such as 200, 404, 500, etc.) under the group to build a multi-dimensional monitoring system; use hash tables to speed up the grouping and counting process;

[0139] Monitor the system's CPU, memory, and I / O usage in real time to assess the current system load. Use machine learning models (such as regression analysis and time series forecasting) to predict the optimal aggregation granularity based on the system load and the number of monitored indicators.

[0140] The system load is evaluated using the following formula:

[0141] SystemLoad=α×f CPU (CPU util ,CPU sat )+β×f MEM (MEM util ,MEM sat )+γ

[0142] ×f IO (IO util ,IO sat )+δ×f RT (RT)+∈×f HLC (HLC)

[0143] Moreover, f X is the load calculation function of resource X, as follows:

[0144]

[0145] f HLC (HLC) = 1 + HLC

[0146] Among them, CPU util is the CPU usage (percentage), MEM util is memory usage (percentage), IO util is I / O usage (percentage), CPU sat is the CPU saturation (work queue length, etc.), MEM sat is memory saturation (page swap frequency, etc.), IO sat is the I / O saturation (I / O request queue length, etc.), RT is the system average response time (milliseconds), HLC is the historical load change coefficient, α, β, γ, δ,∈, are the weight coefficients of each parameter, and α+β+γ+δ+∈=1; CPU threshold ,MEM threshold ,IO threshold , CPU sat_threshold ,MEM sat_threshold ,IO sat_threshold , RT threshold is a predefined threshold used to normalize the input parameters.

[0147] Automatically adjust the aggregation granularity within the time window based on the prediction results. For example, reduce the aggregation granularity during peak load periods to increase data accuracy, and increase the granularity during off-peak periods to reduce resource consumption.

[0148] Utilize the characteristics of time series data (such as periodicity, trend, etc.) to remove redundant or repeated data.

[0149] Combining business rules and statistical methods, it automatically identifies and filters outliers (such as extremely high or low log volumes, unusually frequent HTTP status code changes, etc.); converts processed time series data into monitoring indicators in real time and outputs them to the monitoring system or storage system;

[0150] Regularly evaluate the performance of current time series data generation and processing (such as processing speed, resource utilization, etc.), provide feedback based on the evaluation, and form a closed-loop optimization mechanism.

[0151] The working principle of the above technical solution is as follows: initially, a basic time window size is set according to business needs and the log generation rate. An adaptive algorithm is introduced, which continuously monitors the real-time change rate of log data (such as the speed of increase or decrease in log volume); when a surge or decrease in log volume is detected, the adaptive algorithm dynamically adjusts the size of the time window according to preset rules or the prediction results of the machine learning model. This ensures the timeliness of data processing while avoiding wasting resources when the log volume is small, or causing data backlogs when the log volume surges; to prevent data omissions, an overlapping strategy between time windows is set to ensure that key data points are covered multiple times, thereby improving data integrity and reliability; within each time window, log data is grouped according to the generated aggregation key (based on business category and URL path); efficient data structures such as hash tables are used to speed up the grouping process, ensuring that logs with the same business category and URL path are quickly merged into the same group.

[0152] The number of log entries in each group is counted as the basic monitoring data for the group. While counting, the number of occurrences of different HTTP status codes (such as 200, 404, 500, etc.) in each group is also counted in detail. In this way, a multi-dimensional monitoring system that includes the number of logs and the distribution of HTTP status codes is constructed, providing rich data support for subsequent in-depth analysis. The system's CPU, memory, and I / O usage are monitored in real time to assess the current system load. Based on the system load and the number of monitoring indicators, machine learning models (such as regression analysis and time series prediction) are used to predict the optimal aggregation granularity. Based on the prediction results, the aggregation granularity within the time window is automatically adjusted. For example, during peak load periods, the aggregation granularity is reduced to increase data accuracy, while during off-peak load periods, the granularity is increased to reduce resource consumption. The characteristics of time series data (such as periodicity and trending) are utilized to perform deduplication processing to reduce redundant data. Efficient data compression algorithms (such as differential encoding and run-length encoding) are used to compress processed data to save storage space. Business rules and statistical methods are combined to automatically identify and filter outliers (such as extremely high or low log counts, abnormally frequent HTTP status code changes, etc.) to ensure data accuracy. Processed time series data is converted into monitoring indicators in real time and output to the monitoring system or storage system for subsequent analysis and display. The performance of current time series data generation and processing (such as processing speed and resource utilization) is regularly evaluated, and feedback is provided based on the evaluation results. Through this closed-loop optimization mechanism, the data processing process is continuously optimized to improve overall performance and efficiency.

[0153] The effects of the above technical solution are as follows: by introducing an adaptive algorithm to dynamically adjust the time window size, the technical solution can flexibly respond to data processing needs in different scenarios according to the real-time change rate of monitoring data, ensuring the timeliness and accuracy of data processing; setting an overlapping strategy between time windows effectively prevents data omissions, improves data integrity, and makes monitoring results more reliable; based on real-time evaluation of system load and the number of monitoring indicators, the machine learning model is used to predict and automatically adjust the aggregation granularity, so that data accuracy can be increased during peak load periods and resource consumption can be reduced during low load periods, thereby achieving optimal resource allocation; the grouping and counting process is accelerated through efficient data structures such as hash tables, which significantly reduces CPU and memory consumption and improves overall processing performance; detailed statistical analysis The number of occurrences of different HTTP status codes under the group is used to build a multi-dimensional monitoring system, providing richer and more accurate data support for business analysis; the characteristics of time series data are used for deduplication processing, and an efficient data compression algorithm is adopted to reduce storage space usage. At the same time, business rules and statistical methods are combined to automatically identify and filter outliers, ensuring the accuracy and reliability of time series data; the processed time series data is converted into monitoring indicators in real time and output to the monitoring system or storage system, allowing business personnel to obtain key monitoring information in a timely manner, providing strong support for rapid response and decision-making; the performance of time series data generation and processing is regularly evaluated, and feedback and optimization are provided based on the evaluation results, forming a closed-loop optimization mechanism to continuously improve monitoring efficiency and effectiveness. The construction of a multi-dimensional monitoring system and accurate and reliable time series data provide an important basis for business analysis, performance tuning, troubleshooting, etc., helping enterprises better understand the status of business operations, identify potential problems, and make timely optimizations and improvements. The above complex system load assessment formula comprehensively considers the utilization and saturation of multiple key resources such as CPU, memory, I / O, as well as system response time and historical load changes, providing a comprehensive and integrated assessment of system load. By introducing exponential functions and saturation parameters, the formula can dynamically reflect changes in resource usage. In particular, when resource utilization approaches or exceeds the threshold, it can significantly increase the load assessment value, thereby promptly reflecting potential performance bottlenecks. The introduction of the historical load change coefficient (HLC) enables the formula to predict possible future load trends and provide early warning of potential performance issues. By adjusting the weight coefficient and threshold, the formula can be customized according to the characteristics and needs of different systems, with high flexibility.At the same time, it is also easy to expand and can add more resources and performance indicators; the formula provides quantitative system load assessment results, making performance monitoring and management more accurate and helping to make data-driven decisions; by identifying and quantifying system load, administrators can more effectively allocate resources and optimize performance, thereby improving system efficiency; early warning when the system load is too high helps prevent failures and interruptions caused by system overload; by optimizing resource usage, unnecessary hardware investment is reduced, the utilization rate of existing resources is improved, and operating costs are reduced; keeping the system load within a reasonable range can ensure that users have a good service experience, reduce response time, and improve service reliability; it provides a scientific basis for system administrators and decision makers to help them make decisions about system expansion, maintenance, and upgrades.

[0154] In one embodiment of the present invention, the S5 includes:

[0155] S51. Configure the rule engine to define rules for identifying abnormal logs (such as high error rates, long response times, etc.), and integrate machine learning models such as time series analysis and cluster analysis to automatically discover potential performance issues or abnormal patterns.

[0156] S52. Calculate key performance indicators (KPIs) in real time, including request response time, access volume, and error rate, and update them to the monitoring system;

[0157] S53. Key performance indicators are displayed through the monitoring interface, and based on the alarm mechanism, when KPIs exceed the preset threshold, an alarm is automatically triggered to notify relevant personnel.

[0158] The above technical solution works as follows: A rules engine is configured within the system to define a series of rules for identifying anomalous logs based on business requirements. These rules may cover key performance indicators such as high error rates and long response times. To improve the accuracy and efficiency of anomaly detection, the rules engine integrates machine learning models, such as time series analysis and cluster analysis. These models automatically analyze patterns and trends in log data to identify potential performance issues or abnormal patterns. The rules engine scans log data in real time or periodically, applying defined rules and machine learning models to identify anomalous logs. When an anomaly is detected, the relevant information is recorded and prepared for further processing. The system calculates a series of key performance indicators (KPIs) in real time, including but not limited to request response time, number of visits, and error rate. These indicators are important for evaluating system performance and stability. The calculation process may involve processing, aggregation, and statistics of raw log data to ensure accurate and reliable KPI values. The calculated KPI values are updated in real time to the monitoring system. This allows business personnel and administrators to view system performance indicators in real time through the monitoring system and understand the system's operational status. The monitoring system provides an intuitive interface for displaying key performance indicators (KPIs). These metrics are typically presented in charts and dashboards, allowing users to quickly understand overall system performance and trends. The system employs an alert mechanism that automatically triggers alerts when key performance indicators (KPIs) exceed preset thresholds. These thresholds are set based on business needs and are used to determine whether the system is experiencing abnormal conditions. Alert notifications can be sent via various means, such as email, SMS, and instant messaging, ensuring that relevant personnel receive alert information promptly and take appropriate action.

[0159] The above technical solution achieves the following: By configuring a rules engine and integrating machine learning models, the system can automatically identify abnormal logs and potential performance issues or unusual patterns. This automated detection reduces the need for manual intervention and improves the efficiency and accuracy of anomaly detection. The system calculates key performance indicators (KPIs) in real time and updates them to the monitoring system, providing business personnel and administrators with immediate visibility into the system's operational status. This real-time nature facilitates rapid response to system issues, preventing them from escalating and impacting user experience. The introduction of machine learning models, such as time series analysis and cluster analysis, enables the system to automatically learn and adapt to patterns and trends in log data. This intelligent analysis more accurately identifies potential issues and provides deeper insights. Based on pre-set thresholds and alert mechanisms, when key performance indicators (KPIs) exceed normal ranges, the system automatically triggers alerts to notify relevant personnel. This precise alerting ensures that issues receive timely attention and resolution, reducing potential losses caused by delays. Through real-time monitoring and anomaly detection, the system can promptly identify and address potential performance issues or unusual patterns, thereby improving system stability and reliability. This helps reduce system downtime, enhance user experience, and maintain business continuity. Through real-time monitoring and analysis of key performance indicators (KPIs), system administrators can understand system resource usage and make optimizations and adjustments as needed. This helps better utilize system resources and improve overall operational efficiency. The intuitive display and in-depth analysis provided by the monitoring interface provide important decision-making support for business personnel and administrators. They can use monitoring data to evaluate system performance, develop optimization strategies, and make more informed decisions. When system issues arise, automatically triggered alerts can quickly notify relevant personnel, promoting team collaboration and rapid response. This helps reduce communication costs and response times, and improve team collaboration efficiency.

[0160] In one embodiment of the present invention, the S6 includes:

[0161] S61. Dynamically generate real-time monitoring indicators based on business needs, customize monitoring items and thresholds, and update monitoring data in real time;

[0162] S62. Set the validity period of the monitoring indicators (e.g., N hours), automatically clean up the expired and unupdated indicators, and perform differentiated optimization of the cleaning strategy based on the data access frequency;

[0163] S63. Monitor the performance of the entire log monitoring system, including the load of the Kafka cluster and consumer services. Based on the monitoring results, tune the system, such as adding Kafka partitions and optimizing consumer configuration.

[0164] The working principle of the above technical solution is that the system can dynamically generate real-time monitoring indicators based on specific business needs. This means that the system is no longer limited to fixed monitoring items and can flexibly adapt to business changes. Users can customize monitoring items and corresponding thresholds based on actual needs. This allows the system to more accurately reflect key points in business operations and issue timely alerts when problems arise. The system updates monitoring data in real time, ensuring that users receive the latest and most accurate information. This real-time performance is crucial for promptly identifying and addressing potential issues. Monitoring indicators are assigned an expiration period (e.g., N hours) to ensure that the system does not retain outdated data indefinitely. This helps save storage space and improves system performance. The system automatically cleans up expired monitoring indicators that have not been updated. This process is automated and requires no manual intervention, reducing administrator workload. The system also optimizes cleanup policies based on data access frequency. Frequently accessed data may be retained longer, while less frequently accessed data is cleaned up more quickly. This differentiated optimization strategy helps further improve system efficiency and resource utilization. The system monitors the performance of the entire log monitoring system, including the load of key components such as the Kafka cluster and consumer services. By monitoring the performance indicators of these components in real time, the system can promptly identify potential performance bottlenecks or anomalies. Based on the monitoring results, the system conducts in-depth analysis of the monitoring data to identify the root cause of the performance issue. This analysis may involve statistics, aggregation, and visualization of monitoring data. Based on the monitoring results and analytical conclusions, the system tunes the monitoring system. This may include increasing Kafka partitions to increase throughput and optimizing consumer configuration to balance the load. Through these tuning measures, the system can process log data more efficiently, improving overall performance and stability.

[0165] The above technical solution achieves the following benefits: It dynamically generates real-time monitoring indicators based on business needs, enabling the monitoring system to flexibly respond to business changes. This flexibility ensures the monitoring system remains aligned with business objectives, improving monitoring effectiveness and relevance. It also allows users to customize monitoring items and thresholds, enabling the monitoring system to more accurately reflect key business operations. This personalized setup enhances the adaptability and practicality of the monitoring system. Setting expiration dates for monitoring indicators and automatically purging expired, unupdated indicators helps save storage space and reduce unnecessary resource consumption. This resource optimization strategy helps reduce system operating costs and improve resource utilization. By optimizing purging strategies based on data access frequency, the efficiency and accuracy of resource management are further improved. This intelligent management approach ensures the preservation of critical data while reducing the storage burden of non-critical data. It also monitors the performance of the entire log monitoring system, including the load of key components such as the Kafka cluster and consumer services. This comprehensive monitoring helps promptly identify and resolve performance bottlenecks, ensuring stable system operation. Monitoring results can be used to optimize the system, such as adding Kafka partitions and optimizing consumer configuration. These tuning measures can significantly improve the system's processing power and response speed, enhance user experience and business continuity; real-time monitoring and automatic cleanup mechanisms reduce the need for manual intervention and reduce the workload of operations and maintenance personnel. Operations and maintenance personnel can focus more on solving complex problems and improving operation and maintenance efficiency and quality; through customized monitoring items and threshold settings, the system can more accurately identify potential problems and trigger alarms. This precise alarm mechanism helps operations and maintenance personnel respond and handle problems quickly, reducing the impact of failures on the business; real-time updates of monitoring data ensure that users obtain the latest and most accurate information. This real-time nature helps users keep abreast of business operation status and trend changes, providing strong support for business decision-making; through in-depth analysis and visual display of monitoring data, users can more intuitively understand business operations and potential problems. This insight helps users formulate more scientific and reasonable business strategies and optimization plans.

[0166] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A message queue nginx monitoring indicator solution, characterized in that: The scheme includes: S1. Deploy Filebeat as a log collector on the Nginx server to capture Nginx access logs and error logs in real time. S2 and Filebeat encapsulate the collected log data into messages and push them to the Kafka message queue through the TCP protocol; S3, the consumer pulls log messages from Kafka and parses and processes the Nginx access logs in the messages; S4. Aggregate Request-URLs with the same business meaning using an intelligent aggregation algorithm and summarize them into a monitoring indicator set. S5. Through the built-in rule engine and machine learning algorithm, consumers can further analyze the log data, extract key performance indicators, and update them to the monitoring system in real time; S6. Generate various real-time monitoring indicators based on business needs and system configuration and display them in the monitoring system; and implement a regular monitoring indicator expiration processing mechanism to automatically clean up monitoring indicators that have not been updated within a certain period of time.

2. According to claim 1, a message queue nginx monitoring indicator solution is characterized in that: Said S1 comprises: S11. Configure the Nginx server, install Filebeat on the Nginx server, configure Filebeat, and monitor the Nginx access log and error log through Filebeat. S12. Configure the input module of Filebeat, specify the log file path, and set the buffer and backoff strategies; S13. Configure the output module of Filebeat and send log data to Kafka; S14. Start the Filebeat service to capture new entries in the Nginx log file in real time and perform preliminary data cleaning in Filebeat. S15. Mark the log according to the log type and add metadata, where the metadata includes a timestamp and a server identifier. S16. Parse the log entry using a regular expression or JSON parser to extract key information, including the IP address, request URL, and status code.

3. According to the nginx monitoring indicator solution of a message queue according to claim 1, it is characterized in that Said S2 comprises: S21. Encapsulate the parsed log data into a Kafka message and compress the Kafka message using a compression algorithm; S22. Configure Filebeat to push the encapsulated message to the Kafka cluster using the TCP protocol. During the push process, the error retry mechanism ensures that log data is safely resent when the network fluctuates or the Kafka cluster is temporarily unavailable. S24: Monitor the push process and record key indicators, including push success rate and delay.

4. According to claim 3, a message queue nginx monitoring indicator solution is characterized in that: Said S21 comprises: Extract key fields from the log data parsed by Filebeat, including timestamp, log level, client IP, request method, Request-URL, status code, request size, and response time; According to the Kafka message format requirements, the key fields are structured and stored as key-value pairs or JSON format, and metadata is added to each structured log data; and the integrity and accuracy of the metadata are verified; Use the Kafka client library to create a Kafka message object; encapsulate structured log data and its metadata into the Kafka message body, set the message topic and partition key; and configure the message properties; Compress the Kafka message body using a compression algorithm and evaluate the compression effect.

5. According to claim 3, a message queue nginx monitoring indicator solution is characterized in that: Said S23 comprises: When pushing log data to the Kafka cluster, the status code and response information of the push operation are monitored in real time. Errors are captured and identified using the exception handling mechanism provided by the Kafka client library. Classify errors to determine whether they are retryable or non-retryable errors; If it is a retryable error, it will be retried based on the preset retry strategy, and based on the exponential backoff strategy, the waiting time will be increased between each retry; Before retrying, the integrity and consistency of the log data to be resent are checked. By using Kafka's idempotence or transactional features, duplicate or inconsistent data can be avoided after network partition or Kafka cluster failure recovery. During the resending process, the detailed information of each retry is recorded. For the successfully resent log data, the relevant status information is updated and the relevant systems or personnel are notified. If the maximum number of retries or the total retry time limit is reached and the attempt is still unsuccessful, the failed log data will be recorded in the error log and an alarm will be triggered to notify relevant personnel to intervene; Analyze the reasons for retry failures and adjust the retry strategy or optimize the system configuration based on the analysis results.

6. The nginx monitoring indicator solution for a message queue according to claim 1, characterized in that: Said S3 comprises: S31. Deploy the Kafka consumer service, configure the consumer group, subscription topic, and partition allocation strategy, and perform automatic consumer offset management. S32: The consumer pulls log messages from Kafka, decompresses them, parses each message, and recovers the original Nginx log data and its metadata. S33. Distribute the parsed data to different processing pipelines according to the log type.

7. The nginx monitoring indicator solution for a message queue according to claim 6, characterized in that: The S33 includes: Pull and parse log messages from Kafka, obtain specific fields in the metadata or content of each message and identify the log type, and create a log type enumeration or dictionary; Based on business needs and technical architecture, define multiple processing pipelines, each responsible for processing a specific type of log data; Configure the input, output, processing logic, and required resources of each processing pipeline; Write data distribution logic to route parsed data to the corresponding processing pipeline based on log type. Use load balancing to evenly distribute data to various processing pipelines in high-concurrency situations and process the data asynchronously. Establish a communication mechanism between pipelines to share and communicate data between pipelines through message queues, event buses, or distributed caches; After completing data processing, each processing pipeline generates monitoring indicators in the format of the monitoring platform and reports them to the monitoring platform. An alarm mechanism is introduced to automatically trigger an alarm to notify relevant personnel to intervene when an abnormality occurs in the processing pipeline or performance indicators exceed the threshold.

8. The nginx monitoring indicator solution for a message queue according to claim 1, characterized in that: Said S4 comprises: S41. On the Kafka consumer side, receive Nginx access log data from the Kafka message queue; parse each log data into structured information; S42. Parse the Request-URL in each log and extract the key parts, including the path and query parameters. S43, removing irrelevant parts from the URL and standardizing the URL into a comparable format; S44. Further classify the standardized URLs into different business categories according to preset business rules or regular expressions; generate an aggregation key based on the classified business categories and the standardized URL paths; S45. For each aggregation key, count the number of corresponding log entries as the basic data of the monitoring indicator; and further count the number of occurrences of different HTTP status codes under each aggregation key; S46. Set a time window and aggregate the aggregation key within each time window to generate time series data. Dynamically adjust the aggregation granularity based on the number of monitoring indicators and system load. S47. Regularly clean up aggregated data that has not been updated for a long time, convert the aggregated data into monitoring indicators, and output the monitoring indicators to the monitoring system or storage system.

9. The nginx monitoring indicator solution for a message queue according to claim 1, characterized in that: Said S5 comprises: S51. Configure the rule engine, define rules for identifying abnormal logs, and integrate machine learning models to automatically discover potential performance issues or abnormal patterns. S52. Calculate key performance indicators in real time, including request response time, access volume, and error rate, and update them to the monitoring system; S53. Key performance indicators are displayed through the monitoring interface, and based on the alarm mechanism, when KPIs exceed the preset threshold, an alarm is automatically triggered to notify relevant personnel.

10. The nginx monitoring indicator solution for a message queue according to claim 1, characterized in that: Said S6 comprises: S61. Dynamically generate real-time monitoring indicators based on business needs, customize monitoring items and thresholds, and update monitoring data in real time; S62. Set the validity period of monitoring indicators, automatically clean up expired and unupdated indicators, and perform differentiated optimization of the cleanup strategy based on data access frequency; S63. Monitor the performance of the entire log monitoring system and optimize the system based on the monitoring results.

Images