Multi-cluster monitoring data acquisition and processing method and system

Abstract

The invention discloses a multi-cluster monitoring data acquisition and processing method and a multi-cluster monitoring data acquisition and processing system, and aims to solve the problems of complex deployment and large resource overhead of a traditional architecture in a multi-cluster environment, and realize centralized management of monitoring data of each sub-cluster by deploying a VictoriaMetric instance in a master control cluster. The vmagent modules are deployed in the sub-clusters, node and container index data are actively pulled from various Exporters and pushed to the master control cluster in a remote write-in mode, and the problem that each sub-cluster needs to independently deploy a complete monitoring system is avoided. Uniform query and alarm management of a heterogeneous cluster environment are supported through mechanisms such as label isolation and index standardization, and the method has the characteristics of light deployment, flexible expansion, low cost and the like, is suitable for multiple multi-cluster scenes such as cloud native and edge computing, and remarkably improves the capability of a monitoring system in the aspects of resource use efficiency and cross-cluster observability.

Description

Technical Field

[0001] This invention relates to the field of cloud-native infrastructure monitoring technology, and in particular to a method and system for multi-cluster monitoring data acquisition and processing. Background Technology

[0002] With the increasing adoption of containerization and Kubernetes (an open-source container orchestration platform), enterprises are deploying more and more clusters, and multi-cluster deployments across regions and environments have become the norm. The traditional Prometheus architecture (an open-source monitoring system based on time-series data) deploys a separate instance on each cluster, resulting in wasted resources, fragmented data, and difficulties in unified querying.

[0003] Existing extensions for Prometheus that provide high availability, long-term storage, and global query capabilities (such as Thanos / Cortex) support centralized queries, but they involve numerous components, are complex to deploy, and suffer from overload issues in resource-constrained environments (such as edge clusters). Therefore, a more lightweight, flexible, and easily managed multi-cluster monitoring architecture is needed. Summary of the Invention

[0004] To address the issue in existing monitoring systems where sub-clusters require independent deployment of time-series databases and monitoring system components, this invention provides a method and system for multi-cluster monitoring data acquisition and processing. This system allows sub-clusters to upload their monitoring data to the main control cluster in a lightweight manner, reducing resource consumption and improving system scalability.

[0005] The technical solution adopted in this invention is as follows: A method for collecting and processing multi-cluster monitoring data includes the following steps: Step 1: In each sub-cluster, deploy vmagent and an adapted Exporter as lightweight data collection tools using Helm deployment. Each Exporter in the sub-cluster periodically collects performance metric data from different monitoring targets according to predefined collection logic and reports the data to the main control cluster via vmagent. Step 2: The vmagent deployed in each sub-cluster scrapes performance metrics data from each Exporter via HTTP or Prometheus pull. Through the Kubernetes service discovery mechanism, it dynamically discovers Exporters within the sub-cluster and collects their monitoring data. The vmagent performs unified processing, buffering, and compression on the scraped and collected data, adds a cluster identifier, and then periodically uploads the data to the time-series database VictoriaMetrics in the main control cluster via remote write. Step 3: The VictoriaMetrics instance in the master cluster receives monitoring data from multiple sub-clusters and dynamically adjusts the number of vminsert components written based on the data size of the sub-clusters. Step 4: The vminsert component receives the monitoring data and writes it to multiple vmstorage instances using consistent hashing or round-robin methods. The vmstorage instances persist the time-series data in the monitoring data to disk format locally, perform block merging, compression, and data cleanup operations, and provide a data reading interface for the query layer.

[0006] Furthermore, in step one, the collection cycle is configured through the vmagent's capture configuration file.

[0007] Furthermore, in step two, the periodic triggering conditions for vmagent include at least one of the following: reaching a preset periodic disk flushing time interval; or the data block reaching a preset maximum number of rows or bytes.

[0008] Furthermore, in step two, VictoriaMetrics operates in a distributed deployment manner.

[0009] Furthermore, the vmagent in each sub-cluster sends data to the data entry point of the main control cluster via the asynchronous data push protocol remote write.

[0010] Furthermore, in step four, vminsert scales up or down according to the size of the request volume.

[0011] Furthermore, the vmselect query component uses the Prometheus query language to query monitoring data from each sub-cluster. vmselect queries and displays monitoring data from different sub-clusters using tags. The master cluster controls the number of vmselect query components based on query pressure.

[0012] This invention also provides a multi-cluster monitoring data acquisition and processing system, comprising: The Exporter module, as a lightweight collector on the sub-cluster side, is used to periodically collect performance indicator data from different monitoring targets according to predefined collection logic, and report the data to the main control cluster through vmagent. The vmagent module acts as a data collection agent on the sub-cluster side. It is used to collect performance metric data of each Exporter via HTTP or Prometheus pull, dynamically discover Exporters in the sub-cluster and collect their monitoring data through the Kubernetes service discovery mechanism, uniformly process, buffer and compress the collected data, add cluster identifiers, and then periodically upload the data to the time series database VictoriaMetrics in the main control cluster via remote write. The Load Balancer module, as the data entry point of the master cluster, is used to receive remote write traffic and distribute the load. The vmstorage module is used to persist the time-series data in the monitoring data to a local disk format, perform block merging, compression and data cleaning operations, and provide a data reading interface for the query layer.

[0013] Furthermore, both the Exporter module and the vmagent module are deployed in the sub-cluster using Helm as a workload type DaemonSet or Deployment, and the Exporter module and the vmagent module are located in the same domain name space or in the same network reachable location.

[0014] Furthermore, the Load Balancer module is deployed using a native Kubernetes combination, a standalone load balancer, or cloud hosting.

[0015] The beneficial effects of this invention are: 1. Sub-clusters only need to deploy Exporter and vmagent, without needing to deploy a time-series database, which greatly reduces the resource consumption and deployment difficulty of sub-clusters; 2. By combining vmagent and Exporter, the monitoring system of the sub-cluster is simplified to data collection and reporting, avoiding complex time-series database management and configuration; 3. The system manages data from multiple sub-clusters through a centralized master control cluster, supports flexible expansion, and can handle large-scale cluster environments; 4. VictoriaMetrics in the master cluster provides efficient data storage, compression, and query capabilities, ensuring high performance, scalability, and high availability with multiple backups in large-scale monitoring environments. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a schematic diagram of the overall architecture of the present invention. Detailed Implementation

[0017] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of systems and methods consistent with some aspects of the invention as detailed in the appended claims.

[0018] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0019] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0020] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0021] Figure 1 This is a schematic diagram of the overall architecture of the present invention. Figure 1 As shown, the multi-cluster monitoring data acquisition and processing method provided by the present invention may include the following steps: Step 1: In each sub-cluster, deploy vmagent and an adapted Exporter as lightweight data collection tools using Helm. Each Exporter in the sub-cluster periodically collects performance metrics data from different monitoring targets (such as applications, system resources, etc., covering their respective nodes and containers) according to predefined collection logic, and then reports the data to the main control cluster via vmagent.

[0022] The collection period can be configured through the vmagent scraping configuration file. For example, the scraping period can be set to 50 seconds. Setting the scrape_interval field indicates that the current scraping period is configured globally as 50 seconds. That is, if an Exporter does not have a scrape_interval configured separately, then the Exporter's metrics will be scraped once every 50 seconds. If it is configured separately, then the separately configured scraping period will be used. For example, if scrape_interval=15 seconds is configured separately, then the Exporter's metrics will be scraped once every 15 seconds.

[0023] Helm is the package management tool for the open-source container orchestration platform Kubernetes; vmagent is a lightweight data collection agent in the VictoriaMetrics time-series database ecosystem; and Exporter is middleware for monitoring data. By deploying only Exporter and vmagent without deploying the time-series database on the sub-cluster side, the deployment cost of sub-cluster monitoring is significantly reduced. This deployment method avoids the complexity and resource consumption issues associated with deploying a local time-series database on each sub-cluster.

[0024] Step 2: The vmagent deployed in each sub-cluster scrapes the metric data of each Exporter via HTTP or Prometheus pull, and dynamically discovers the Exporters in the sub-cluster and collects their monitoring data through the Kubernetes service discovery mechanism. Then, the vmagent processes, buffers and compresses the scraped and collected data in a unified manner, adds a cluster identifier, and then periodically uploads the data to the time-series database VictoriaMetrics in the main control cluster via remote write.

[0025] There are two situations where vmagent is triggered periodically: (1) the set periodic disk flushing time interval is reached; (2) the data block reaches the set maximum number of rows or the maximum number of bytes. These parameters are the startup parameters configured when vmagent is started in this invention.

[0026] VictoriaMetrics operates in a distributed deployment mode, featuring high availability, high compression ratio, high scalability, and low resource consumption.

[0027] Step 3: The vmagents in each sub-cluster send data to the Load Balancer, the data entry point of the main control cluster, via the asynchronous data push protocol remote write. This avoids directly exposing multiple vmagents to the internal components of VictoriaMetrics, effectively improving the system's decoupling and load balancing capabilities. Subsequently, the VictoriaMetrics instances in the main control cluster receive this monitoring data from multiple sub-clusters and can dynamically adjust the number of vminserts (write services) written based on the data size of the sub-clusters, achieving convenient and efficient data processing.

[0028] vminsert is the write layer component in VictoriaMetrics, responsible for receiving external time series data, performing preliminary processing, and distributing / writing the data to the storage node vmstorage. vminsert is deployed as a Deployment in a Kubernetes cluster. If the resource utilization of vminsert exceeds the tolerance range in a sub-cluster, the number of Deployment replicas can be increased. Deployments are used to efficiently manage the deployment, updates, and scaling of stateless applications.

[0029] In this invention, the master cluster serves as a centralized storage and query system, providing efficient data storage, compression, and query capabilities, and is able to handle large-scale data from multiple sub-clusters. VictoriaMetrics enables efficient storage and query operations in large-scale environments, ensuring efficient data management.

[0030] Step 4: The vminsert in the master cluster receives monitoring data from each sub-cluster and writes the data to multiple vmstorage instances using consistent hashing or round-robin methods. vmstorage, as a data storage component, is a persistent storage node in the cluster. It is responsible for receiving time-series data forwarded by vminsert, persisting it locally in an efficient disk format, performing block merging / compression and data cleanup, and providing a data read interface for the query layer. Here, vminsert can scale up or down according to the request volume to avoid system crashes caused by high load on a single node.

[0031] The VictoriaMetrics suite in the master cluster offers rich query capabilities. Users or third-party platforms can use the Prometheus query language (PromQL) through the query component vmselect to query monitoring data from each sub-cluster. vmselect, in particular, queries and displays monitoring data from different sub-clusters based on tags.

[0032] In this invention, the master control cluster is not only responsible for data storage but also supports efficient data analysis and visualization. The master control cluster can control the number of query components (vmselect) based on query pressure to avoid blocking. This invention enables cross-cluster metric integration, visual analysis, and alarm configuration.

[0033] This invention provides a multi-cluster monitoring data acquisition and processing system, comprising: The Exporter module, acting as a lightweight collector on the sub-cluster side (including cluster resource collectors, network collectors, kubelet collectors, cluster node collectors, GPU collectors, and sensor collectors), aims to periodically collect raw metrics from monitored targets and expose these metrics via the / metrics interface in Prometheus, HTTP, or custom formats. Its primary responsibility is to scrape various metrics from applications, containers, systems, GPUs, BMC (the core component of server hardware management), etc., according to predefined collection logic, and expose the Prometheus / metrics interface for vmagent to scrape. The module's input is the monitored target (including endpoints, ports, and authentication information), and its output is the / metrics HTTP endpoint (in Prometheus format). For example, key configurations might include a list of scraped targets, enabled collection plugins / modules, credentials, and scraping frequency (typically controlled by vmagent's scrape configuration). For deployment, Helm is used to deploy it within each sub-cluster as a DaemonSet or Deployment workload, and it must be in the same domain name space or network reachable location as vmagent. Observable metrics / alarms include data acquisition latency, data acquisition failure rate, response code distribution, and number of BMC access failures. It's important to note that deploying a full time-series database in each sub-cluster is not recommended; deploying only the Exporter can significantly reduce resource consumption.

[0034] The vmagent module acts as a data collection agent (i.e., a collector) on the sub-cluster side. Its purpose is to centrally scrape various Exporter metrics within the sub-cluster, perform unified processing, buffering, and compression, and then remotely write them to the main control cluster. Its main responsibilities include dynamically discovering Exporters based on the Kubernetes service discovery mechanism, scraping Exporters according to the scrape configuration cycle (using Prometheus pull or HTTP), performing unified processing on the scraped samples (such as label normalization / supplementation, aggregation, and deduplication strategies), buffering, compressing, and segmenting, and triggering remote writing according to policies (conditions include reaching the regular disk flush interval, reaching row / byte thresholds, etc.), and finally asynchronously pushing the data to the main control entry point (LoadBalancer) via the remote write protocol. The module's inputs are the / metrics scraping results, service discovery information, and the vmagent configuration file; the output is a remote_write data stream to the Load Balancer. For deployment, vmagent is deployed in each sub-cluster using Helm (as a Deployment or DaemonSet), with centralized configuration and management. This module is scalable and can be adapted to throughput by adjusting the periodic disk flushing interval, row / byte threshold, and parallelism parameters.

[0035] The Load Balancer module, serving as the data entry point for the master cluster, aims to receive remote_write requests from multiple vmagents and distribute traffic to vminsert instances, preventing internal components from being directly exposed to the external network. Its primary responsibilities include receiving remote write traffic and distributing it across the load (using methods such as round-robin, least connections, header-based routing, or path-based routing), while providing entry point protection such as TLS (Transport Layer Security), authentication, and rate limiting to maintain high availability and support horizontal scaling. The module's input is remote_write requests from the sub-cluster vmagents, and its output is forwarding to several vminsert instances. For deployment, it can use a native Kubernetes combination (Service + Ingress), a standalone load balancer (L4 / L7 load balancer), or cloud-hosted (Cloud Load Balancer).

[0036] The vmstorage module's primary purpose is to persist time-series data, perform merging / compression, and provide a query / read interface. Its main responsibilities include receiving data forwarded by vminsert and persisting it to local disk using an efficient block format, performing background tasks such as compression and deleting expired data, and providing a data read interface for vmselect. The module's input is time-series data blocks forwarded by vminsert, and its output is data stored on disk and responding to query requests (via vmselect). For deployment, either a StatefulSet or Deployment workload (depending on requirements) can be used, along with high-IO storage, capacity planning, and backup strategies.

[0037] In terms of configuration management, helm parameters or ConfigMaps (configuration mapping tables) are used uniformly to manage the configurations of vmagent, exporter, etc. In terms of security, this covers TLS, authentication (IP authentication / Token authentication / MTLS mutual certificate authentication), network policies, and BMC / Exporter credential management (using Secrets, which are API objects specifically used to store and manage sensitive information).

[0038] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0039] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for acquiring and processing multi-cluster monitoring data, characterized in that, Includes the following steps: Step 1: In each sub-cluster, deploy vmagent and an adapted Exporter as lightweight data collection tools using Helm deployment. Each Exporter in the sub-cluster periodically collects performance metric data from different monitoring targets according to predefined collection logic and reports the data to the main control cluster via vmagent. Step 2: The vmagent deployed in each sub-cluster scrapes performance metrics data from each Exporter via HTTP or Prometheus pull. Through the Kubernetes service discovery mechanism, it dynamically discovers Exporters within the sub-cluster and collects their monitoring data. The vmagent performs unified processing, buffering, and compression on the scraped and collected data, adds a cluster identifier, and then periodically uploads the data to the time-series database VictoriaMetrics in the main control cluster via remote write. Step 3: The VictoriaMetrics instance in the master cluster receives monitoring data from multiple sub-clusters and dynamically adjusts the number of vminsert components written based on the data size of the sub-clusters. Step 4: The vminsert component receives the monitoring data and writes it to multiple vmstorage instances using consistent hashing or round-robin methods. The vmstorage instances persist the time-series data in the monitoring data to disk format locally, perform block merging, compression, and data cleanup operations, and provide a data reading interface for the query layer.

2. The multi-cluster monitoring data acquisition and processing method according to claim 1, characterized in that, In step one, the collection cycle is configured through the vmagent capture configuration file.

3. The multi-cluster monitoring data acquisition and processing method according to claim 1, characterized in that, In step two, the periodic triggering conditions for vmagent include at least one of the following: reaching the preset periodic disk flushing time interval; the data block reaching the preset maximum number of rows or the maximum number of bytes.

4. The multi-cluster monitoring data acquisition and processing method according to claim 1, characterized in that, In step two, VictoriaMetrics runs in a distributed deployment.

5. The multi-cluster monitoring data acquisition and processing method according to claim 1, characterized in that, The vmagent in each sub-cluster sends data to the data entry point of the main control cluster via the asynchronous data push protocol remote write.

6. The multi-cluster monitoring data acquisition and processing method according to claim 1, characterized in that, In step four, vminsert scales up or down according to the size of the request volume.

7. The multi-cluster monitoring data acquisition and processing method according to claim 1, characterized in that, The vmselect query component uses the Prometheus query language to query monitoring data from each sub-cluster. vmselect queries and displays monitoring data from different sub-clusters using tags. The master cluster controls the number of vmselect components based on query pressure.

8. A multi-cluster monitoring data acquisition and processing system, characterized in that, include: The Exporter module, as a lightweight collector on the sub-cluster side, is used to periodically collect performance indicator data from different monitoring targets according to predefined collection logic, and report the data to the main control cluster through vmagent. The vmagent module acts as a data collection agent on the sub-cluster side. It is used to collect performance metric data of each Exporter through HTTP or Prometheuspull data scraping modes. Through the Kubernetes service discovery mechanism, it dynamically discovers Exporters in the sub-cluster and collects their monitoring data. It performs unified processing, buffering and compression on the scraped and collected data, adds cluster identifiers, and then periodically uploads the data to the time-series database VictoriaMetrics in the main control cluster via remote write. The Load Balancer module, as the data entry point of the master cluster, is used to receive remote write traffic and distribute the load. The vmstorage module is used to persist the time-series data in the monitoring data to a local disk format, perform block merging, compression and data cleaning operations, and provide a data reading interface for the query layer.

9. The multi-cluster monitoring data acquisition and processing system according to claim 8, characterized in that, Both the Exporter module and the vmagent module are deployed in the sub-cluster using Helm as a workload type DaemonSet or Deployment, and the Exporter module and the vmagent module are located in the same domain name space or in the same network reachable location.

10. The multi-cluster monitoring data acquisition and processing system according to claim 8, characterized in that, The LoadBalancer module can be deployed using Kubernetes native combinations, standalone load balancers, or cloud hosting.

Images