Integrated multi-dimensional monitoring and unified alarm system and method

By using tools such as HertzBeat to unify multi-dimensional monitoring and alarm systems, the problems of fragmented monitoring dimensions and chaotic alarm management are solved. It provides a clear and efficient monitoring architecture and intelligent alarm processing flow, improving the operation and maintenance efficiency and system reliability in the cloud-native microservice environment.

CN121125560APending Publication Date: 2025-12-12HAINA ZHIYUAN DIGITAL TECH (SHANGHAI) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511007625.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-07-22
Publication Date
2025-12-12

AI Technical Summary

Technical Problem

Existing technologies suffer from fragmented monitoring dimensions, high integration and maintenance costs, and chaotic alarm management, making it particularly difficult to achieve a global view and efficient operation and maintenance under a cloud-native microservice architecture.

Method used

HertzBeat is used as the unified alert aggregation center, combined with tools such as Kubernetes, Logstash, Elasticsearch, Prometheus, Grafana, Doris, and Sentry to achieve seamless integration of service availability, logs, middleware, and business monitoring. Through unified alert aggregation and intelligent distribution, a closed-loop process from alert generation to real-time delivery is built.

Benefits of technology

It provides a single-pane view of system health, reducing system complexity and maintenance costs, improving the accuracy and speed of fault response, and adapting to the dynamic nature of cloud-native environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121125560A_ABST
    Figure CN121125560A_ABST
Patent Text Reader

Abstract

The invention discloses an integrated multi-dimensional monitoring and unified alarm system and method. The system comprises a service availability monitoring module, a log monitoring module, a middleware monitoring module, a business monitoring module and a unified alarm aggregation and intelligent distribution module. The system carries out automatic discovery and availability detection on the service instances in the Kubernetes through HertzBeat; the method comprises the following steps of: acquiring, analyzing and alarming a log through Logstash, Elasticsearch and Grafana; the index monitoring and the alarm of the middleware are realized through Prometheus and Grafana; and real-time analysis and alarm of service data are realized through Dores and Grafana. And all alarms are uniformly aggregated by HertzBeat, forwarded to Sentry for intelligent distribution, and finally pushed to an enterprise communication tool in real time through Webhook. According to the invention, comprehensive and integrated monitoring of service availability, logs, middleware and businesses is realized, an automatic closed-loop process from alarm generation to accurate reaching is constructed, and the operation and maintenance efficiency and the system reliability in a cloud native environment are significantly improved.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer information technology and network operation and maintenance, and particularly relates to a monitoring system and method for IT infrastructure, especially a system and method capable of integrating multi-dimensional monitoring of service availability, logs, middleware and business indicators, and realizing unified alarm processing and real-time distribution, and particularly suitable for modern microservice architecture based on containerization technologies such as Kubernetes. BACKGROUND

[0002] With the rise of cloud computing and microservice architecture, modern application systems are split into numerous independent deployment service units that communicate through the network and run in a dynamic and elastic containerized environment (such as Kubernetes). This distributed architecture, while improving system flexibility and scalability, also brings unprecedented challenges to system stability assurance and rapid fault localization.

[0003] The existing monitoring solutions in the prior art usually have one or more of the following problems:

[0004] 1. Monitoring dimensions are fragmented: There are various monitoring tools on the market, but most of them focus on a single field, for example, Prometheus is good at monitoring indicators, ELK (Elasticsearch, Logstash, Kibana) is good at log analysis, and availability monitoring and business monitoring may require other specialized tools. This results in the need for the operation and maintenance team to maintain multiple heterogeneous monitoring systems, and the monitoring data is scattered in different platforms, making it difficult to form a global and correlated view of the system health status.

[0005] 2. High integration and maintenance costs: Integrating multiple independent monitoring tools into a system that works together requires a lot of development and configuration work, and the conversion of data formats, the unified management of permissions, and subsequent maintenance and upgrades are extremely complex and require a lot of manpower and resources.

[0006] 3. Alarm management is chaotic: Due to the dispersion of monitoring systems and the large number of alarm sources, "alarm storms" are easily generated. Alarm information formats are not uniform, and there is a lack of unified convergence, noise reduction, deduplication and intelligent routing mechanisms. Alarm notification channels are scattered, and operation and maintenance personnel are overwhelmed by the massive amount of alarm information, resulting in low response efficiency, and the truly critical fault signals are easily ignored.

[0007] Therefore, there is an urgent need for a new monitoring and alarm solution that can natively integrate multiple core monitoring dimensions, simplify the system architecture, and provide a closed-loop management process from alarm generation, aggregation, intelligent distribution to real-time reach, to meet the higher requirements for operation and maintenance efficiency and system reliability in the cloud-native era. SUMMARY

[0008] The main objective of this invention is to overcome the shortcomings of the prior art and provide a multi-dimensional monitoring and real-time alarm system and method with a clear architecture, comprehensive functions, and unified process to solve the technical pain points of scattered monitoring data and chaotic alarm management.

[0009] Definitions:

[0010] 1. HertzBeat: An open-source real-time monitoring and alerting tool with powerful custom monitoring capabilities and agentless functionality. In this invention, it serves as both the core of service availability monitoring and the center of unified alert aggregation.

[0011] 2. Kubernetes (K8s): An open-source system for automatically deploying, scaling, and managing containerized applications.

[0012] 3. Pod: The smallest deployable computing unit that can be created and managed in Kubernetes.

[0013] 4. Logstash: An open-source server-side data processing pipeline that can collect data from multiple sources, transform the data, and then send the data to your favorite "repository".

[0014] 5. Elasticsearch: A distributed, RESTful search and data analytics engine.

[0015] 6. Grafana: An open-source, cross-platform data visualization and monitoring tool, commonly used to display time-series data.

[0016] 7. Prometheus: An open-source system monitoring and alerting toolkit, particularly suitable for recording purely numerical time-series data.

[0017] 8. Doris: An open-source, real-time analytical database based on an MPP architecture.

[0018] 9. Sentry: A bug tracking tool that helps developers monitor and fix crashes in real time. It can be used for event management and alert assignment.

[0019] 10. Webhook: A concept in web development that allows an application to send real-time information to other applications when a specific event occurs. It is often used for inter-system linkage and notification.

[0020] 11. Enterprise WeChat Bot / DingTalk Bot: Refers to robot programs provided by the Enterprise WeChat and DingTalk platforms, which can receive API calls and automatically send messages in specified group chats. They are an important tool for enterprise automated notifications.

[0021] To achieve the above object, the application adopts the following technical scheme: an integrated multi-dimensional monitoring and real-time alarm system, the system comprises:

[0022] A service availability monitoring module is used for health state detection of service instances (Pods) running in a containerized environment; the module automatically discovers and acquires the list of all service instances by actively calling the API interface of a container orchestration platform (such as Kubernetes), and performs periodic survival detection on the service instances by using TCP or HTTP protocol, and generates an availability alarm when the detection fails.

[0023] A log monitoring module is used for centralized management and analysis of logs generated by service instances; the module comprises a log collection unit (such as Logstash) for collecting logs from each service instance at a regular time, a log storage and indexing unit (such as Elasticsearch) for storing massive log data, and a log analysis and alarm unit (such as Grafana) for querying and analyzing log data according to preset rules (such as keyword matching, log frequency) and generating a log alarm when the conditions are met.

[0024] A middleware monitoring module is used for monitoring the performance indicators of various types of middleware (such as Redis, Zookeeper, Doris, Kafka, etc.) relied on by the system; each middleware instance reports its internal state indicator data to a time series database (such as Prometheus) for storage; a data analysis and alarm unit (such as Grafana) queries the indicators in the time series database and judges according to preset threshold rules, and generates a middleware alarm when the indicators exceed the normal range.

[0025] A business monitoring module is used for real-time monitoring of core business indicators of the application system; business data is transmitted to an analytical database (such as Doris) in real time or quasi-real time; a data analysis and alarm unit (such as Grafana) performs real-time aggregation analysis on the business data in the analytical database and judges according to the preset business health threshold, and generates a business alarm when the indicators are abnormal.

[0026] A unified alarm aggregation and intelligent distribution module serves as the alarm processing center of the system and comprises:

[0027] 1. An alarm aggregation center (preferably HertzBeat) is provided, which provides an API interface for receiving and integrating various alarms generated by the log, middleware and business monitoring modules.

[0028] 2. The alarm aggregation center processes the external alarms received and the service availability alarms generated by itself.

[0029] 3、The integrated alarm is forwarded to an event management platform (such as Sentry), which automatically groups, de-duplicates, and assigns personnel to the alarm event according to preset routing rules and scheduling strategies.

[0030] 4、The event management platform pushes notifications containing alarm details and handler information to one or more enterprise instant messaging tool robots (such as Qiyi Bot and Dingding Bot) in real time through a Webhook mechanism, thereby achieving alarm closure, real-time, and precise reach.

[0031] The application also provides a monitoring and alarm method based on the above system, comprising the following steps:

[0032] Step (1) Perform availability detection on service instances running in a containerized environment, and generate an availability alarm if the detection fails;

[0033] Step (2) Analyze logs generated by the service instances, and generate a log alarm if the preset log rules are met;

[0034] Step (3) Monitor the performance indicators of the middleware associated with the service instances, and generate a middleware alarm if the preset threshold rules are met;

[0035] Step (4) Perform real-time analysis on the business indicators of the application system, and generate a business alarm if the preset business threshold is met;

[0036] Step (5) Unified alarm aggregation of the availability alarm, log alarm, middleware alarm, and business alarm generated in steps (1), (2), (3), and (4) to form a standardized alarm event stream;

[0037] Step (6) Automatically assign handlers to the alarm event stream, and push alarm information to instant messaging tools in real time through a Webhook mechanism.

[0038] Further, the unified alarm aggregation in step (5) is that an alarm aggregation center HertzBeat receives alarms generated by other analysis tools Grafana and integrates them with the availability alarms generated by itself.

[0039] Further, the automatic handler assignment in step (6) is that the alarm event stream is forwarded to an event management platform Sentry, which assigns personnel according to preset rules.

[0040] Specifically, a monitoring and alarm method based on the above system comprises the following steps:

[0041] Step 1, automatic discovery and availability detection: periodically call the Kubernetes API to obtain the metadata of all Pods in the cluster, and perform preset TCP or HTTP health checks on each Pod. If the check fails, generate an availability alarm and go to step 5.

[0042] Step 2, log analysis and alarm: use Logstash to collect service logs and store them in Elasticsearch; Grafana queries Elasticsearch according to log rules. If the rule is triggered, a log alarm is generated and goes to step 5.

[0043] Step 3, middleware metric monitoring and alarm: use Prometheus to pull performance metrics from each middleware instance; Grafana queries Prometheus according to threshold rules. If the metric exceeds the threshold, a middleware alarm is generated and goes to step 5.

[0044] Step 4, business data monitoring and alarm: receive real-time business data and store it in Doris database; Grafana analyzes the data in Doris in real time. If the business metric is not up to standard, a business alarm is generated and goes to step 5.

[0045] Step 5, unified alarm aggregation: send the alarms generated by Grafana in steps 2, 3, and 4 to HertzBeat through API interface; HertzBeat integrates these external alarms with the availability alarms generated in step 1.

[0046] Step 6, intelligent distribution and real-time notification: HertzBeat forwards the integrated alarms to Sentry; Sentry analyzes the alarm events and automatically assigns handlers; Sentry calls the interface of Qiyi Bot or Dingding Bot through Webhook to send alarm information to the assigned personnel in real time.

[0047] Compared with the prior art, the beneficial effects of the present application are:

[0048] 1. Comprehensive monitoring view: The present application seamlessly integrates infrastructure availability, system logs, middleware performance, and core business metrics into a "single-pane" view of system health for operations personnel, effectively breaking down data silos.

[0049] 2. Clear and efficient architecture: The system architecture is clear and each component has a clear responsibility. By using HertzBeat to balance the roles of availability monitoring and alarm aggregation, multiple monitoring sources are unified, reducing system complexity and maintenance costs.

[0050] 3. Intelligent alarm processing flow: A full-automated alarm processing closed loop from generation, aggregation, intelligent distribution to real-time notification is created. In particular, Sentry is introduced for alarm routing and personnel assignment, which significantly reduces human intervention, reduces alarm noise, and improves the accuracy and speed of fault response.

[0051] 4. High automation and cloud-native friendly: Service availability monitoring can automatically discover newly online services without manual configuration, perfectly adapting to the dynamics of Kubernetes environment. The entire monitoring system has no code intrusion to business services, and is easy to deploy and expand. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is the system architecture and data flow block diagram according to an embodiment of the present application.

[0053] Figure 2 is the system architecture and data flow detailed block diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] The technical solutions of the present application will be further described in detail below in combination with the drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it. The embodiments do not limit the present application. The present application is described in a typical Kubernetes-based microservice environment.

[0055] Referring to Figure 1 , Figure 2 The system of the present application is composed of four monitoring modules and a unified alarm module.

[0056] 1. Availability monitoring:

[0057] The core component is HertzBeat. HertzBeat periodically calls the pod list interface of K8sAPI through the configured Kubernetes credentials to automatically obtain the IP, port and other information of Pods in all namespaces. This mechanism realizes the automatic discovery of new Pods. Then, HertzBeat performs a preset TCP or HTTP ping request on each Pod. If the request times out or returns an unexpected status code, HertzBeat determines that the Pod is unavailable, generates an availability alarm immediately, and puts the alarm into its internal alarm queue.

[0058] 2. Log monitoring:

[0059] Service logs are collected from each node by Logstash instances deployed as DaemonSet, filtered and formatted, and then sent to the Elasticsearch cluster for storage. In Grafana, operations personnel create dashboards and alarm rules, such as "Query the number of logs with log level ERROR and containing the keyword "NullPointerException" in the past 5 minutes, and if it is greater than 3, trigger an alarm". The alarm engine of Grafana periodically executes this query, and once the condition is met, a log alarm is generated.

[0060] 3. Middleware monitoring:

[0061] Redis, Zookeeper, Doris, Kafka, and other middleware in the cluster have configured PrometheusExporter to expose their internal performance indicators (such as connection number, queue length, GC frequency, etc.). The Prometheus service periodically pulls indicator data from these Exporters. In Grafana, alarm rules are created based on the Prometheus data source, such as "The number of ISR (In-Sync Replicas) of a certain Kafka instance is less than the total number of partition replicas". When this PromQL query returns a result, Grafana generates a middleware alarm.

[0062] 4. Business monitoring:

[0063] When processing key business processes, application services write core indicators (such as the number of successful payments per minute, user registration volume) to the Doris database in real time. Grafana directly connects to Doris, performs real-time analysis through SQL queries, and sets alarm rules, such as "The average query success rate in the past 10 minutes is less than 99.5%". If the condition is met, Grafana generates a business alarm.

[0064] 5. Unified alarm aggregation and intelligent distribution:

[0065] This is the key innovative process of the present application.

[0066] (1) Step A - Alarm aggregation: In the notification channels (Notification Channels) of Grafana, configure a channel pointing to the HertzBeat Webhook API. All alarms generated by Grafana (logs, middleware, business) will be sent to this interface.

[0067] (2) Step B - Alarm integration: After receiving the alarm from Grafana, HertzBeat uniformly formats it with the availability alarm generated by itself, forming a standardized alarm event stream.

[0068] (3) Step C - Intelligent distribution: HertzBeat forwards the unified alarm event stream to Sentry. In Sentry, rich rules can be configured, such as "database-related alarms are automatically assigned to the DBA team" and "P1-level alarms are immediately notified to all core developers", based on which Sentry completes the automatic personnel distribution of alarm events.

[0069] (3) Step D - Real-time reach: After assigning personnel, Sentry triggers a webhook pointing to the enterprise internal collaboration platform. The webhook calls the API of the WeChat Bot or DingDing Bot, sends the formatted alarm card (including alarm summary, related indicators, chart links, responsible person, etc.) to the specified group chat, and realizes real-time notification of alarms.

[0070] The present application aims to solve the problem of fragmented monitoring dimensions and chaotic alarm management in the prior art. The system includes a service availability monitoring module, a log monitoring module, a middleware monitoring module, a business monitoring module, and a unified alarm aggregation and intelligent distribution module. The system automatically discovers and detects the availability of service instances in Kubernetes through HertzBeat; collects, analyzes and alarms logs through Logstash, Elasticsearch and Grafana; monitors and alarms middleware indicators through Prometheus and Grafana; realizes real-time analysis and alarm of business data through Doris and Grafana. After all alarms are unified by HertzBeat, they are forwarded to Sentry for intelligent distribution, and finally pushed to enterprise communication tools through webhook in real time. The present application realizes comprehensive and integrated monitoring of service availability, logs, middleware and business, and builds an automatic closed-loop process from alarm generation to accurate reach, significantly improving the operation and maintenance efficiency and system reliability in the cloud native environment.

[0071] The above is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. An integrated multi-dimensional monitoring and unified alarm system, characterized in that, Comprise: a service availability monitoring module for health state detection of service instances running in a containerized environment and generating availability alarms when detection fails; a log monitoring module for analysis of logs generated by the service instances and generating log alarms when preset log rules are met; a middleware monitoring module for monitoring performance indicators of middleware associated with the service instances and generating middleware alarms when indicators meet preset threshold rules; a business monitoring module for real-time analysis of business indicators of an application system and generating business alarms when business indicators meet preset thresholds; a unified alarm aggregation and intelligent distribution module for receiving and integrating the availability alarms, log alarms, middleware alarms and business alarms, forming a unified alarm event stream, and forwarding the alarm event stream to an event management platform for automatic allocation of processing personnel, and finally pushing alarm information to designated instant messaging tools in real time through a Webhook mechanism.

2. The integrated multi-dimensional monitoring and unified alarm system of claim 1, wherein: The service availability monitoring module automatically discovers service instances by calling an application programming interface (API) of a container orchestration platform and performs survival detection using TCP or HTTP protocols.

3. The integrated multi-dimensional monitoring and unified alarm system of claim 1, wherein: The unified alarm aggregation and intelligent distribution module includes an alarm aggregation center that provides an API interface to receive alarms generated from other monitoring modules and integrates all alarms with availability alarms generated by itself.

4. The integrated multi-dimensional monitoring and unified alarm system of claim 3, wherein: The alarm aggregation center is HertzBeat, and the event management platform is Sentry.

5. The integrated multi-dimensional monitoring and unified alarm system of claim 1, wherein: The log monitoring module includes Logstash for collecting logs, Elasticsearch for storing logs, and Grafana for analyzing and generating alarms.

6. The integrated multi-dimensional monitoring and unified alarm system of claim 1, wherein: The middleware monitoring module includes a time series database (Prometheus) for collecting indicators and Grafana for analyzing and generating alarms.

7. The integrated multi-dimensional monitoring and unified alarm system of claim 1, wherein: The business monitoring module includes an analytical database (Doris) for storing business data and Grafana for analyzing and generating alarms.

8. The integrated multi-dimensional monitoring and unified alarming system according to any one of claims 1-7, wherein an integrated multi-dimensional monitoring and unified alarming method is implemented. The steps include: Step (1) performing availability detection on service instances running in a containerized environment and generating availability alarms if detection fails; Step (2) analyzing logs generated by the service instances and generating log alarms if preset log rules are met; Step (3) monitoring performance indicators of middleware associated with the service instances and generating middleware alarms if preset threshold rules are met; Step (4) performing real-time analysis of business indicators of an application system and generating business alarms if preset business thresholds are met; Step (5) uniformly aggregating availability alarms, log alarms, middleware alarms and business alarms generated in steps (1), (2), (3) and (4) to form a standardized alarm event stream; Step (6) automatically assigns the processing personnel to the alarm event stream and pushes the alarm information to instant messaging tools in real time through a Webhook mechanism.

9. The integrated multi-dimensional monitoring and unified alarm method of claim 8, wherein: The unified alarm aggregation in step (5) is that an alarm aggregation center HertzBeat receives alarms generated by other analysis tools Grafana and integrates the availability alarms generated by itself.

10. The integrated multi-dimensional monitoring and unified alarm method of claim 8, wherein: The automated processing personnel assignment in step (6) is that the alarm event stream is forwarded to an event management platform Sentry, and personnel are assigned according to preset rules by the event management platform.