Monitoring platform data summarization method, system, equipment and medium

Abstract

The invention provides a monitoring platform data summarization method, system and device and a medium, and belongs to the technical field of computers. The method comprises the following steps: deploying a Kafka cluster, and using the Kafka cluster as middleware for receiving data from different Prometheus instances; configuring an adapter for each instance, capturing monitoring data of the Prometheus instance through the adapter, and pushing the monitoring data to a Kafka cluster; reading the monitoring data in the Kafka cluster by using a stream processing framework, and storing the monitoring data in a time sequence database after standardization processing; obtaining monitoring data from the time sequence database through the front-end application interface, and performing visual display; and when the monitoring data triggers a preset alarm condition, automatically generating an alarm notification. By integrating the Kafka message middleware, the streaming processing engine and the special time sequence database, real-time aggregation, standardized management and intelligent analysis of multi-source monitoring data are realized.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and more particularly relates to a monitoring platform data aggregation method, system, device and medium. Background Art

[0002] In the modern information technology sector, as enterprises accelerate their digital transformation, business systems are becoming increasingly complex and diverse. To ensure high availability and optimized performance of these systems, more and more enterprises are adopting monitoring and alerting platforms to provide real-time visibility into their IT infrastructure and services. Prometheus, an open-source system monitoring and alerting tool suite, has been widely adopted by many enterprises due to its powerful data model, flexible query language (PromQL), and efficient storage mechanisms.

[0003] However, as business grows and projects increase, many companies choose to deploy Prometheus monitoring in different projects. However, this also brings the following new challenges: Data dispersion: Each Prometheus instance runs independently, resulting in monitoring data being scattered across different locations, making unified management and analysis difficult.

[0004] Lack of integration: Since there is no direct data sharing mechanism between different Prometheus instances, operations and maintenance personnel need to switch between multiple platforms to obtain a complete monitoring view, which increases operational complexity and time costs.

[0005] Difficulty in alarm management: The alarm information generated by each Prometheus instance is also independent of each other, which makes centralized management and priority sorting more difficult and makes it easy to miss important alarm notifications.

[0006] Visualization limitations: Existing visualization tools can usually only display data from a single Prometheus instance and cannot provide a global perspective to help operations teams quickly identify the root cause of the problem. Summary of the Invention

[0007] In response to the above problems, the purpose of the present invention is to provide a monitoring platform data aggregation method, system, equipment and medium. By integrating Kafka message middleware, streaming processing engine and dedicated time series database, real-time aggregation, standardized management and intelligent analysis of multi-source monitoring data are realized, which significantly improves the comprehensiveness, timeliness and operation and maintenance response efficiency of monitoring.

[0008] To achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions: In a first aspect, an embodiment of the present application provides a monitoring platform data aggregation method, comprising: Deploy a Kafka cluster and use it as middleware to receive data from different Prometheus instances; Configure an adapter for each Prometheus instance, use the adapter to capture the monitoring data of the Prometheus instance and push it to the Kafka cluster; Use the stream processing framework to read monitoring data from the Kafka cluster, perform normalization processing, and then store it in a time series database; Obtain monitoring data from the time series database through the front-end application interface and display it visually; When the monitoring data triggers the preset alarm conditions, an alarm notification is automatically generated.

[0009] In an optional embodiment, deploying a Kafka cluster as middleware to receive data from different Prometheus instances includes: Start the Zookeeper server and the Kafka server, and create a Topic named prometheus_data, which will be used as a middleware to store monitoring data from different Prometheus instances.

[0010] In an optional embodiment, deploying a Kafka cluster as middleware to receive data from different Prometheus instances includes: Start the Zookeeper server and the Kafka server, and create a Topic named prometheus_data, which will be used as a middleware to store monitoring data from different Prometheus instances.

[0011] In an optional embodiment, the method of using a stream processing framework to read monitoring data from a Kafka cluster, performing normalization processing, and storing the data in a time series database includes: Use Apache Flink or Spark Streaming to read monitoring data from the Kafka cluster; Convert monitoring data into a unified data format and perform standardized processing; The standardized monitoring data is stored in the time series database.

[0012] In an optional embodiment, obtaining monitoring data from a time series database through a front-end application interface and visually displaying the data includes: Launch a front-end page developed based on React or Vue.js, and obtain monitoring data from the time series database through the RESTful API; Generate corresponding charts or dashboards based on monitoring data and display them.

[0013] In an optional embodiment, when the monitoring data triggers a preset alarm condition, automatically generating an alarm notification includes: Set preset alarm conditions and alarm rules; When monitoring data triggers an alarm condition, an alarm notification is generated; According to the alarm rules, the alarm notification is sent to the enterprise WeChat or email service through the Webhook interface.

[0014] In an optional embodiment, the time series database adopts InfluxDB or TimescaleDB.

[0015] In a second aspect, an embodiment of the present application further provides a monitoring platform data aggregation system, comprising: The deployment module is used to deploy a Kafka cluster and use it as middleware to receive data from different Prometheus instances; The data collection module is used to configure an adapter for each Prometheus instance, capture the monitoring data of the Prometheus instance through the adapter, and push it to the Kafka cluster; The data processing module is used to read the monitoring data in the Kafka cluster using the stream processing framework, perform normalization processing, and store it in the time series database; The information display module is used to obtain monitoring data from the time series database through the front-end application interface and display it visually; The alarm module is used to automatically generate an alarm notification when the monitoring data triggers the preset alarm conditions.

[0016] In a third aspect, an embodiment of the present application further provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the steps of the monitoring platform data aggregation method as described in any one of the above items are implemented.

[0017] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the monitoring platform data aggregation method as described in any one of the above items are implemented.

[0018] It can be seen from the above technical solutions that the present invention has the following advantages: In the monitoring platform data aggregation method provided in this application, Kafka is deployed as the data transmission hub, and prometheus-kafka-adapter is used to import data from multiple Prometheus instances into Kafka, and then stored in a time series database after standardized processing. At the same time, real-time stream processing technology is used to conduct in-depth analysis of the data, and the monitoring results are displayed through the Web application front-end. In addition, this method also provides flexible alarm management functions, supports multiple notification methods, and effectively improves the enterprise's IT operation and maintenance efficiency and response speed. This solution not only solves the problem of difficulty in centralized management and analysis of multi-source heterogeneous monitoring data, but also enhances the maintainability and practicality of the system.

[0019] This application deploys the prometheus-kafka-adapter to automatically retrieve monitoring data from multiple Prometheus instances. This allows for regular, up-to-date monitoring information without manual intervention, significantly reducing the workload for operations and maintenance personnel. This automated process reduces the risk of manual errors, ensures data accuracy and consistency, and improves overall operational efficiency.

[0020] This application aggregates monitoring data from various Prometheus instances into a central platform, providing a global view of the data. This allows operations teams to gain a comprehensive understanding of the status of the entire IT infrastructure, quickly identify the source of problems, and effectively manage and optimize resources. This application standardizes data from different Prometheus instances, ensuring that all data is consistent in format and structure, facilitating subsequent analysis and storage.

[0021] Based on the results of real-time data analysis, this application can quickly identify potential problems and trigger corresponding alarm notifications, helping operation and maintenance personnel take timely action to prevent the problem from further escalating.

[0022] This application utilizes a web-based user interface that supports various data display formats, including charts and dashboards. The intuitive interface design allows users to access complex cluster information through simple operations, lowering the barrier to entry. The front-end page not only displays static data but also supports various interactive features, such as data filtering and sorting, allowing users to analyze monitoring data in depth based on their needs and accelerate problem resolution.

[0023] This application supports sending alert notifications via multiple channels, including WeChat and email, ensuring that critical alerts are promptly communicated to relevant personnel. Furthermore, different alert strategies and notification methods can be configured based on actual needs, enhancing the system's flexibility and practicality. Furthermore, this application allows users to customize alert conditions based on their business needs, such as threshold-based alerts or early warnings based on predictive models, improving the accuracy and effectiveness of alerts.

[0024] This application is suitable for businesses and organizations of all sizes, from large internet companies to small and medium-sized enterprises, enabling efficient IT operations and management. It is also applicable to diverse industries, such as finance, healthcare, and manufacturing, providing users with versatile and powerful tools to meet diverse information management needs. The system architecture of this application is designed to be flexible and easily scalable, allowing for the addition of additional functional modules or integration of other third-party services based on future business development needs, maintaining technological advancement and competitiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A flow chart of the monitoring platform data aggregation method provided in this application.

[0027] Figure 2 This is a structural diagram of the monitoring platform data aggregation system provided in this application.

[0028] Figure 3 This is a schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0029] The various embodiments of the present disclosure will be described in more detail below in the specific steps of the monitoring platform data aggregation method. The present disclosure can have various embodiments, and adjustments and changes can be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but rather that the present disclosure should be understood to cover all adjustments, equivalents, and / or alternatives that fall within the spirit and scope of the various embodiments of the present disclosure.

[0030] Hereinafter, the terms "include" or "may include" as used in various embodiments of the present disclosure indicate the presence of disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. In addition, as used in various embodiments of the present disclosure, the terms "include," "have," and their cognates are intended only to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing, and should not be understood as excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing.

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0032] See also Figure 1 FIG. 1 is a flow chart of a method for aggregating monitoring platform data in a specific embodiment, the method comprising: S1: Deploy a Kafka cluster and use it as middleware to receive data from different Prometheus instances.

[0033] In the specific implementation, first, you need to deploy a high-availability Kafka cluster. This cluster will serve as the middleware to receive data from different Prometheus instances. Deploying the Kafka cluster involves starting the Zookeeper server and the Kafka server, and creating a Topic named prometheus_data to store monitoring data.

[0034] For example, this step aims to build a highly available and scalable message middleware by deploying a Kafka cluster to centrally receive and temporarily store monitoring data from multiple sources. First, plan the number of cluster nodes based on the business scale (typically 3-5 physical or virtual machines), ensuring inter-node network connectivity and resource redundancy (e.g., CPU, memory, and storage to meet high-concurrency data writing requirements). Second, deploy Zookeeper as Kafka's metadata management center and configure a cluster mode (e.g., a 3-node pseudo-distributed or standalone cluster) to ensure metadata consistency and availability. Next, deploy the Kafka service on each node and build a cluster architecture that supports data redundancy and load balancing by configuring parameters such as broker.id (node ​​unique identifier), replication.factor (number of replicas, typically ≥ 2), and num.partitions (number of partitions to increase parallel consumption). Finally, create a topic named prometheus_data and implement partitioning and replication strategies to ensure orderly storage and highly reliable transmission of monitoring data, providing a foundation for subsequent data aggregation across multiple Prometheus instances.

[0035] S2: Configure an adapter for each Prometheus instance, use the adapter to capture the monitoring data of the Prometheus instance and push it to the Kafka cluster.

[0036] In a specific implementation, a prometheus-kafka-adapter adapter is configured for each Prometheus instance. This adapter will scrape data from the Prometheus instance and push it to the Kafka cluster. The adapter configuration file must specify the Kafka address, topic, and address of the Prometheus instance to be scraped.

[0037] For example, this step uses an adapter to connect the Prometheus instance to the Kafka cluster. The core is to build a bridge link for "data capture-format conversion-message push". During implementation, a prometheus-kafka-adapter is deployed independently for each Prometheus instance (such as monitoring nodes distributed in different business clusters or regions). The service address of the Kafka cluster (such as bootstrap.servers), the target topic (prometheus_data), and the address of the Prometheus instance to be crawled (such as http: / / prom1.example.com:9090) are specified in the configuration file. The adapter actively crawls monitoring indicators (such as CPU usage and memory usage) based on Prometheus's HTTP API, converts the raw data into a Kafka-compatible message format (such as JSON or Protobuf), and writes the data to the Kafka cluster through an asynchronous push mechanism. By configuring the scrape_interval (scraping interval) to synchronize with Prometheus's own collection cycle, data timeliness is ensured. At the same time, it supports dynamic adjustment of adapter parameters (such as the number of concurrent connections and retry strategy) to adapt to the load differences of different instances, achieving stable aggregation of multi-source data.

[0038] S3: Uses a stream processing framework to read monitoring data from the Kafka cluster, normalizes it, and stores it in a time series database.

[0039] In a specific implementation, a stream processing framework (such as Apache Flink or Spark Streaming) is used to read data from the Kafka cluster, normalize it, and then store it in a time series database. Normalization involves converting data in different formats into a unified format for subsequent analysis and storage. The processed data is then written to a high-performance time series database, such as InfluxDB or TimescaleDB. These databases are specifically designed for time series data and can efficiently store and query large amounts of monitoring data, facilitating fast query and visualization.

[0040] For example, this step uses a stream processing framework to process and standardize monitoring data in real time, providing high-quality data in a unified format for subsequent analysis and storage. For implementation, choose a stream processing framework that supports high throughput and low latency (such as Apache Flink or Spark Streaming), subscribe to the prometheus_data topic through the Kafka consumer interface, and read real-time data in streaming mode.

[0041] The processing logic includes: 1. Format standardization, converting the unstructured / semi-structured data of different Prometheus instances (such as differences in label naming and inconsistent timestamp formats) into a unified structured model (such as a JSON object with fixed fields); 2. Anomaly filtering, eliminating invalid or dirty data through preset rules (such as value out-of-bounds and timestamp timeout); 3. Dimension supplementation, adding contextual tags to the data based on metadata (such as the business line to which the instance belongs and the deployment region).

[0042] The processed data is written to the database through the time series database adapter component (such as InfluxDB's LineProtocol or TimescaleDB's PostgreSQL interface), leveraging the database's time index optimization (such as timestamp-based partitioning and compression) and efficient query capabilities (such as aggregation by time range) to ensure efficient storage and access of massive monitoring data.

[0043] S4: Obtain monitoring data from the time series database through the front-end application interface and display it visually.

[0044] In a specific implementation, a web application based on a modern front-end framework is developed. This application retrieves data from a time series database via a RESTful API and presents it to users in the form of charts, dashboards, and other formats, providing an intuitive and user-friendly front-end interface. This visual interface helps operations personnel quickly understand the health status of the system and make appropriate adjustments.

[0045] Among them, the front-end framework uses React or Vue.js to develop intuitive and friendly front-end pages to display monitoring data and alarm information.

[0046] In addition, script commands are automatically executed regularly to ensure that the data on the front-end page is always up to date, improving the real-time performance and accuracy of the system.

[0047] For example, this step uses a front-end application to visually present monitoring data. The core focus is on building a user interaction chain from "data acquisition - visual rendering - real-time updates." Specifically, a web application is developed based on a modern front-end framework (such as React or Vue.js). This interface connects to a time series database via a RESTful API to retrieve monitoring data on demand (e.g., for specified metrics and time ranges). The visualization layer uses charting libraries (such as ECharts and Recharts) to convert data into line charts, bar charts, and dashboards. Information density is enhanced through a layered design (e.g., a main view displays core metrics, subviews drill down to details) and interactive features (e.g., time range filtering and metric comparison). To ensure real-time data availability, the front-end obtains the latest data through scheduled polling (e.g., triggering an API request every 30 seconds) or a WebSocket persistent connection mechanism, dynamically refreshing charts. Furthermore, an integrated alarm information display module (e.g., a red-highlighted notification banner) is integrated. Combined with responsive design (adapting to PC and mobile screens) and accessibility optimizations (e.g., high-contrast color schemes and keyboard navigation support), this provides operations and maintenance personnel with an efficient and user-friendly monitoring interface.

[0048] S5: When the monitoring data triggers the preset alarm condition, an alarm notification is automatically generated.

[0049] In a specific implementation, preset alarm conditions and alarm rules are first set; when monitoring data triggers an alarm condition, an alarm notification is generated; then, based on the alarm rules, the alarm notification is sent to the enterprise WeChat or email service via the Webhook interface. Specifically, alarm rules are set and alarm notifications are sent to the enterprise WeChat or email service via the Webhook interface. When monitoring data triggers the preset alarm condition, the system automatically generates an alarm notification and sends it to the enterprise WeChat or other notification channels via the Webhook interface. In addition, different notification methods can be set according to actual needs, such as email, SMS, etc., to ensure that critical alarms are conveyed to relevant personnel in a timely manner.

[0050] For example, this step implements intelligent alerting for monitoring data through pre-set rules. The core concept is to build a closed-loop mechanism: condition detection, notification generation, and multi-channel push notifications. Specifically, configure alert rules in a stream processing framework or independent alerting engine (such as PrometheusAlertManager). These rules include trigger conditions (e.g., "CPU usage > 90% for 5 minutes"), severity levels (e.g., warning, emergency), and associated tags (e.g., instance, business line). When processed monitoring data matches these rules, the system automatically generates alert notifications (including the metric value, trigger time, and contextual description) and routes them to pre-set channels through an extensible notification engine. Notification channels support webhook interfaces (e.g., WeChat Work bots, Lark group chats), email services (SMTP protocol), or SMS gateways (HTTP API). Configuring the priority of different channels (e.g., emergency alerts prioritize SMS, general alerts via WeChat Work) ensures timely delivery of critical information. Furthermore, support for alert suppression (e.g., merging notifications when the same metric is triggered repeatedly) and recovery notifications (e.g., sending a closed message when a metric returns to normal) improves the accuracy and maintainability of alert management.

[0051] In this embodiment, by building a Kafka cluster as a high-availability data transfer layer, combining with adapters to achieve unified access to multi-source Prometheus data, using the stream processing framework to complete real-time standardized processing and store it in a proprietary time series database, and finally through front-end visualization and intelligent alarm mechanism, effectively solve the problems of multi-source monitoring data integration difficulties, low storage and query efficiency, and delayed alarm response, and significantly improve the scalability, data consistency and operation and maintenance response efficiency of the monitoring system, while ensuring the timeliness and accuracy of alarm reach, providing efficient and reliable monitoring guarantee for the stable operation of large-scale distributed systems.

[0052] like Figure 2 As shown, the following is an embodiment of the monitoring platform data aggregation system provided by the embodiment of the present disclosure. The system and the monitoring platform data aggregation method of the above-mentioned embodiments belong to the same inventive concept. For details not fully described in the embodiment of the monitoring platform data aggregation system, please refer to the embodiment of the above-mentioned monitoring platform data aggregation method.

[0053] A monitoring platform data aggregation system includes: a deployment module, a data acquisition module, a data processing module, an information display module and an alarm module.

[0054] The deployment module is used to deploy a Kafka cluster and use it as middleware to receive data from different Prometheus instances.

[0055] The data collection module is used to configure an adapter for each Prometheus instance, capture the monitoring data of the Prometheus instance through the adapter, and push it to the Kafka cluster.

[0056] The data processing module is used to read the monitoring data in the Kafka cluster using the stream processing framework, perform standardization processing, and then store it in the time series database.

[0057] The information display module is used to obtain monitoring data from the time series database through the front-end application interface and display it visually.

[0058] The alarm module is used to automatically generate an alarm notification when the monitoring data triggers the preset alarm conditions.

[0059] The monitoring platform data aggregation system provided in this embodiment implements full-link processing of monitoring data through a layered architecture: first, a Kafka cluster is deployed to build a high-availability data bus to take over multi-source Prometheus data buffering; then, an adapter is configured to complete the connection between heterogeneous data sources and message queues; then, a stream processing engine is used to implement real-time data cleaning, format unification, and standardized distribution; finally, the storage structure is optimized through a time series database, and combined with front-end visualization and intelligent alarm mechanisms, a complete closed loop from data collection, cleaning, storage to display is formed, ensuring that the monitoring system has high throughput, low latency, strong scalability, and real-time response capabilities.

[0060] Figure 3 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0061] The monitoring platform data aggregation method provided in the embodiment of the present application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiment of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange components differently. In the embodiment of the present invention, the electronic device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0062] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, a button, a camera, a display, and a SIM card interface, etc.

[0063] A processor may include one or more processing units, such as a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0064] The processor can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on the instruction opcode and timing signal to complete the control of instruction fetching and execution.

[0065] The processor may also include a memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or is reusing. If the processor needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0066] The external memory interface can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of an electronic device. The external memory card communicates with the processor through the external memory interface, enabling data storage. For example, files such as music and videos can be stored on the external memory card.

[0067] Internal memory can be used to store computer-executable program code, which includes instructions. The processor executes the instructions stored in the internal memory to perform various functional applications and data processing of the electronic device. The internal memory can include a program storage area and a data storage area. The internal memory can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0068] The wireless communication function of an electronic device can be implemented through an antenna, a wireless communication module, a modem processor, and a baseband processor.

[0069] Wireless communication modules can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0070] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0071] Electronic devices can achieve shooting functions through ISP, camera, video codec, GPU, display and application processor.

[0072] Electronic devices can achieve display functions through GPU, display screen and application processor.

[0073] A GPU is a microprocessor for image processing that connects the display screen to the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0074] The display screen is used to display images, videos, etc. The display screen includes a display panel.

[0075] The above-mentioned electronic equipment implements the data aggregation method of the monitoring platform of this application by deploying a Kafka cluster to build a high-availability data bus and configuring an adapter to achieve unified access to multi-source Prometheus data. It completes real-time data cleaning and format standardization through a stream processing framework and then stores it in a time series database. It ensures real-time data display through a front-end visual interface and an automatic refresh mechanism, and achieves rapid response to anomalies through intelligent alarm rules and multi-channel notifications, ultimately achieving the beneficial effect of building a high-throughput, low-latency, and highly scalable monitoring platform, significantly improving data aggregation efficiency, analysis accuracy, and operation and maintenance response capabilities.

[0076] The storage medium provided in this application stores a program product that can implement the monitoring platform data aggregation method.

[0077] Monitoring platform data aggregation methods include: Deploy a Kafka cluster and use it as middleware to receive data from different Prometheus instances; Configure an adapter for each Prometheus instance, use the adapter to capture the monitoring data of the Prometheus instance and push it to the Kafka cluster; Use the stream processing framework to read monitoring data from the Kafka cluster, perform normalization processing, and then store it in a time series database; Obtain monitoring data from the time series database through the front-end application interface and display it visually; When the monitoring data triggers the preset alarm conditions, an alarm notification is automatically generated.

[0078] In some possible implementations, the monitoring platform data aggregation method disclosed herein can be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps described in the above "Exemplary Method" section of this specification according to various exemplary implementations of the present disclosure.

[0079] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0080] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A monitoring platform data aggregation method, characterized in that: include: Deploy a Kafka cluster and use it as middleware to receive data from different Prometheus instances; Configure an adapter for each Prometheus instance, use the adapter to capture the monitoring data of the Prometheus instance and push it to the Kafka cluster; Use the stream processing framework to read monitoring data from the Kafka cluster, perform normalization processing, and then store it in a time series database; Obtain monitoring data from the time series database through the front-end application interface and display it visually; When the monitoring data triggers the preset alarm conditions, an alarm notification is automatically generated.

2. The monitoring platform data aggregation method according to claim 1, characterized in that: The deployment of a Kafka cluster as middleware to receive data from different Prometheus instances includes: Start the Zookeeper server and the Kafka server, and create a Topic named prometheus_data, which will be used as a middleware to store monitoring data from different Prometheus instances.

3. The monitoring platform data aggregation method according to claim 2, characterized in that: The adapter is configured for each Prometheus instance, and the monitoring data of the Prometheus instance is captured by the adapter and pushed to the Kafka cluster, including: Configure the prometheus-kafka-adapter adapter for each Prometheus instance; In the adapter configuration file, specify the Kafka cluster address, topic, and Prometheus instance address to be scraped.

4. The monitoring platform data aggregation method according to claim 3, characterized in that: The use of the stream processing framework to read monitoring data from the Kafka cluster, perform normalization processing, and store it in the time series database includes: Use Apache Flink or Spark Streaming to read monitoring data from the Kafka cluster; Convert monitoring data into a unified data format and perform standardized processing; The standardized monitoring data is stored in the time series database.

5. The monitoring platform data aggregation method according to claim 4, characterized in that: The monitoring data is obtained from the time series database through the front-end application interface and visualized, including: Launch a front-end page developed based on React or Vue.js, and obtain monitoring data from the time series database through the RESTful API; Generate corresponding charts or dashboards based on monitoring data and display them.

6. The monitoring platform data aggregation method according to claim 5, characterized in that: When the monitoring data triggers a preset alarm condition, an alarm notification is automatically generated, including: Set preset alarm conditions and alarm rules; When monitoring data triggers an alarm condition, an alarm notification is generated; According to the alarm rules, the alarm notification is sent to the enterprise WeChat or email service through the Webhook interface.

7. The monitoring platform data aggregation method according to claim 4, characterized in that: The time series database uses InfluxDB or TimescaleDB.

8. A monitoring platform data aggregation system, characterized in that: The system adopts the monitoring platform data aggregation method according to any one of claims 1 to 7; The system comprises: The deployment module is used to deploy a Kafka cluster and use it as middleware to receive data from different Prometheus instances; The data collection module is used to configure an adapter for each Prometheus instance, capture the monitoring data of the Prometheus instance through the adapter, and push it to the Kafka cluster; The data processing module is used to read the monitoring data in the Kafka cluster using the stream processing framework, perform normalization processing, and store it in the time series database; The information display module is used to obtain monitoring data from the time series database through the front-end application interface and display it visually; The alarm module is used to automatically generate an alarm notification when the monitoring data triggers the preset alarm conditions.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the monitoring platform data aggregation method according to any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the monitoring platform data aggregation method according to any one of claims 1 to 7 are implemented.

Images