Prometheus-based dynamic label setting method and system for monitoring system

Abstract

The invention relates to the technical field of big data platform monitoring, in particular to a prometheus-based dynamic label setting method and system for a monitoring system, and the method comprises the following steps: in a Prometheus configuration file, carrying out the filtering, renaming and adding operation on a label obtained from a Kubernetes API (Application Program Interface) by utilizing a relatablconfiguration item, so as to achieve the dynamic setting of monitoring labels of service and pod in Kubernetes; writing and executing a shell script through a Shell-Operator tool, monitoring a Kubernetes event, and dynamically setting or modifying a monitoring label for the Pod according to the label and node information of the Pod; the prometheus-based monitoring system dynamic label setting method and system provided by the invention have the beneficial effects that monitoring requirements in different scenes are flexibly adapted, and codes or deployed application programs do not need to be modified; the monitoring requirement of finer granularity can be met; detailed information of a specific scene or event can be retrieved; the requirement of more accurate alarm of specific monitoring data can be met; and rapid positioning and source tracing can be provided for troubleshooting and diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of big data platform monitoring, and in particular to a method and system for dynamically setting labels for a monitoring system based on Prometheus. Background Art

[0002] With the rapid development of digital, intelligent and platform technologies, a complex situation has emerged in the PaaS platform, with new and old systems coexisting and containerized and non-containerized clusters running in parallel. Therefore, the demand for monitoring data collection and access to monitoring systems has become diversified, requiring monitoring data to be accurately labeled according to source, content and scope.

[0003] However, the traditional setting method is often to develop and add settings by the monitored service system itself, or to modify the configuration of prometheus and monitoring tools through tedious manual operations, which not only increases the development workload, but also makes the modification process cumbersome and inefficient. For system platforms that have been deployed or built, in order to ensure the stability of the service and reduce operation and maintenance costs, it is usually not easy to modify the original system for monitoring access. What is more complicated is that due to the characteristics of containers, service pods often drift dynamically in nodes or different pods of services need to be set with different labels, which makes the management of service labels extremely difficult. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for dynamically setting labels for a monitoring system based on Prometheus to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for dynamically setting labels for a monitoring system based on Prometheus, the method comprising the following steps:

[0006] In the Prometheus configuration file, use the relabel_configs configuration item to filter, rename, and add labels obtained from the Kubernetes API to dynamically set monitoring labels for services and pods in Kubernetes.

[0007] Use the Shell-Operator tool to write and execute shell scripts, monitor Kubernetes events, and dynamically set or modify monitoring labels for Pods based on Pod labels and node information, ensuring that Prometheus effectively collects and classifies monitoring data based on labels.

[0008] In the Consul service registration and discovery system, users are allowed to set custom monitoring labels for services through the interface or API, and Prometheus automatically discovers and monitors the corresponding services from Consul based on the labels;

[0009] In the YAML definition of Helm Charts, set monitoring-related labels and annotations for Kubernetes resources so that Prometheus can identify and collect monitoring data for these resources;

[0010] In the Exporter configuration, specify the monitoring indicators to be collected and add custom labels to them. The monitoring data with labels is sent to the Prometheus server through Exporter to achieve refined classification and management of monitoring data.

[0011] Preferably, the relabel_configs configuration item is used in the Prometheus configuration file to modify the labels obtained from the Kubernetes API, including but not limited to filtering unnecessary labels, renaming labels, and adding new monitoring labels.

[0012] Preferably, the Shell-Operator tool monitors Kubernetes' Pod creation, update or deletion events, combines the Pod's labels and node information, executes a custom shell script, dynamically sets or modifies annotations and labels for the Pod, and then implements fine monitoring of the Pod in Prometheus.

[0013] Preferably, in the Consul service registration and discovery system, the user sets a custom monitoring tag for the service through an interface or API, and the tag is read by Prometheus as part of the service registration information during service discovery and applied to the selection of monitoring targets and data collection.

[0014] Preferably, when defining Kubernetes resources, the Helm Charts adds monitoring-related configurations to metadata.labels and metadata.annotations of the resources, so that Prometheus automatically discovers and collects monitoring data of the resources based on the configuration, thereby realizing dynamic label setting and classification management of the monitoring data.

[0015] A prometheus-based monitoring system dynamically sets labels, the system comprising:

[0016] A Prometheus monitoring system to collect and process monitoring data;

[0017] Configuration module for setting the external_labels property in Prometheus so that these labels are automatically added to all metrics when querying or exporting data;

[0018] Job configuration module, which is used to divide different Prometheus jobs according to the type of monitoring target, so as to achieve refined division of monitoring data;

[0019] Kubernetes integration module, which automatically discovers services and pods in the cluster through Kubernetes' Service Discovery mechanism and relabel_configs configuration item, and allows filtering, renaming, adding and modifying labels obtained from the Kubernetes API;

[0020] Shell-Operator integration module, which monitors Kubernetes events and dynamically sets or modifies Pod monitoring labels through shell scripts based on event types and Pod attributes;

[0021] Consul integration module, which allows users to set custom monitoring labels for services in Consul through the interface or API. Prometheus performs service discovery and monitoring based on these labels;

[0022] Helm Charts integration module, which allows setting monitoring-related labels and annotations for Kubernetes resources in Charts YAML definitions;

[0023] The Exporter configuration module is used to specify monitoring indicators in the Exporter configuration and add custom labels so that the monitoring data with labels can be sent to the Prometheus server.

[0024] Preferably, the Kubernetes integration module further includes:

[0025] Use metadata.labels to add monitoring labels in the Pod's YAML definition;

[0026] Use metadata.annotations to add monitoring-related metadata to the Pod's YAML definition;

[0027] In the Prometheus configuration file, use the relabel_configs configuration item to filter, rename, add, and modify labels obtained from the Kubernetes API.

[0028] Preferably, the Shell-Operator integrated module further comprises:

[0029] Write a shell script to monitor events of Pods in a specified namespace in Kubernetes;

[0030] Dynamically set or modify monitoring labels and annotations for Pods based on their current labels, node information, and event types;

[0031] Identify the services that need to be monitored by checking the labels and annotations of the Pods, and add necessary monitoring labels to them so that Prometheus can automatically discover and monitor them.

[0032] Preferably, the Consul integration module further includes:

[0033] Allow users to set custom monitoring tags for services in Consul through the interface or API;

[0034] Tags are stored as part of the service registration information and queried through the Consul API;

[0035] Prometheus uses Consul as a service discovery source and filters and selects services to be monitored based on labels.

[0036] Preferably, the Exporter configuration module further includes:

[0037] Specify the monitoring indicators to be collected in the Exporter configuration and add custom labels to them;

[0038] Tags are sent to the Prometheus server as part of the monitoring data, enabling refined classification and management of monitoring data;

[0039] Different Exporters collect monitoring data for different applications or services and add unique labels to the data for query and analysis in Prometheus.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The method and system for dynamically setting labels for a monitoring system based on Prometheus proposed in the present invention can flexibly adapt to monitoring requirements in different scenarios without modifying the code or deployed applications; can meet more fine-grained monitoring requirements; can meet the retrieval of detailed information on specific scenarios or events; can meet the requirements for more accurate alarms for specific monitoring data; can provide faster positioning and tracing for troubleshooting and diagnosis; has strong scalability, and can easily add new labels to describe monitored elements. In short, dynamically setting monitoring labels brings higher flexibility, fine-grainedness, accuracy and scalability to the monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of the method of the present invention;

[0043] Figure 2 This is a system architecture diagram of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] For example, see Figure 1 The present invention provides a technical solution: a method for dynamically setting labels for a monitoring system based on Prometheus, the method comprising the following steps:

[0046] 1) Kubernetes label settings

[0047] Prometheus is integrated with Kubernetes. Prometheus can automatically discover services and pods in the cluster and collect their monitoring data through Kubernetes' ServiceDiscovery mechanism. In the Prometheus configuration file (the default file prometheus.yml), use relabel_configs to filter and modify the labels obtained from the Kubernetes API.

[0048] Use metadata.labels to add monitoring labels

[0049] For example, when defining a Kubernetes service, add product-related labels in the metadata.labels section of the YAML file:

[0050] apiVersion:v1

[0051] kind:Service

[0052] metadata:

[0053] name:my-service

[0054] labels:

[0055] app:my-app

[0056] product:my-product

[0057] spec:

[0058] #...Other configurations...

[0059] Use metadata.annotations to add monitoring metadata

[0060] Similarly, we can use metadata.annotations in the Pod's YAML definition to add monitoring-related metadata, for example:

[0061] apiVersion:v1

[0062] kind:Pod

[0063] metadata:

[0064] name:my-pod

[0065] annotations:

[0066] prometheus.io / scrape: "true"

[0067] prometheus.io / path:" / metrics"

[0068] prometheus.io / port:"9090"

[0069] spec:

[0070] #...Other configurations...

[0071] 2) Shell-Operator tag settings

[0072] Shell-operator is a tool for Kubernetes and other cloud-native environments that allows users to define and execute custom Kubernetes event response logic through shell scripts (or other executable files). Shell-Operator can listen to Kubernetes events and trigger the execution of corresponding shell scripts. These scripts can decide how to set or modify monitoring labels based on the event type (such as Pod creation, update, or deletion) and Pod properties (such as labels). The shell script written in this solution can listen to kubernetes events about pods in specified namespaces. Set labels for telegraf or cadvisor pods in kubernetes, and you can view the indicator information with labels when viewing monitoring indicators in prometheus.

[0073] Automatic label setting

[0074] The Shell-Operator script can query the Kubernetes API to obtain the current labels and node information of the Pod, and then dynamically set or modify the monitoring labels for the Pod based on this information without affecting the business. Through the node labels and custom labels, annotations and labels are set for the corresponding pod.

[0075] Tags and service identification

[0076] The Shell-Operator script can check the labels and annotations of the Pod, identify the services that need to be monitored, and add the necessary monitoring labels to them. These labels can be used for Prometheus's Service Discovery so that Prometheus can automatically discover and monitor these services, and then dynamically convert them to the required labels according to the pre-configured rules of Prometheus.

[0077] 3) Consul tag settings

[0078] Service discovery and configuration management. Consul allows service registration and discovery, and can be integrated with Prometheus. In Consul, services can store configuration information, including custom monitoring tags, through key-value (KV) storage.

[0079] Custom tags and configuration

[0080] In this solution, a monitoring center service is written, part of which is access monitoring. This service allows users to add services to Consul through the interface or API, set custom monitoring tags for the services, and view and edit monitoring access information. These tags will be registered as part of the service and can be queried through Consul's API. Prometheus can use Consul as a service discovery source and filter and select services to be monitored based on these tags.

[0081] 4)Charts tag settings

[0082] Declarative application management, Helm Charts allows us to define and manage Kubernetes applications in a declarative way. In Charts, we can set labels and annotations for Kubernetes resources (such as service, deployment, etc.).

[0083] Setting monitoring labels in Charts

[0084] In the YAML definition of Charts, we can add monitoring-related labels to the metadata.labels of the service, for example:

[0085]

[0086] At the same time, we can also add monitoring-related annotations to the metadata.annotations of the deployment or statefulset so that Prometheus can identify and collect monitoring data for these resources.

[0087] 5) Monitoring Exporter tag settings

[0088] Exporter is a key component in the Prometheus ecosystem, responsible for collecting monitoring data from various applications and services. Exporter usually exposes monitoring data in a format that Prometheus can understand.

[0089] Setting tags in Exporter

[0090] When configuring an Exporter (such as Telegraf), we can specify the monitoring metrics to be collected and add custom labels to them. These labels will be sent to the Prometheus server as part of the monitoring data.

[0091] For example, in the Telegraf configuration file, we can add a custom tag for a plugin:

[0092] [[inputs.cpu]]

[0093] #...Other configurations...

[0094] [inputs.cpu.tags]

[0095] environment="production"

[0096] zone="us-east-1"

[0097] Component-level tag management

[0098] By setting labels at the Exporter level, we can achieve more refined classification and management of monitoring data. Different Exporters can collect monitoring data for different applications or services and add unique labels to this data for query and analysis in Prometheus.

[0099] In this way, a complete set of monitoring label setting solutions is formed based on Prometheus, combined with kubernetes, consul, shell-operator, chart, and telegraf in the monitoring data collection process. It helps users to quickly classify and manage monitoring data, improve the efficiency of troubleshooting and locating problems, improve operation and maintenance efficiency and service quality, and facilitate users to uniformly manage and filter monitoring data.

[0100] Embodiment 2, based on embodiment 1, proposes a prometheus-based monitoring system dynamic label setting system, the system comprising:

[0101] A Prometheus monitoring system to collect and process monitoring data;

[0102] Configuration module for setting the external_labels property in Prometheus so that these labels are automatically added to all metrics when querying or exporting data;

[0103] Job configuration module, which is used to divide different Prometheus jobs according to the type of monitoring target, so as to achieve refined division of monitoring data;

[0104] The Kubernetes integration module automatically discovers services and pods in the cluster through Kubernetes' Service Discovery mechanism and the relabel_configs configuration item, and allows filtering, renaming, adding, and modifying labels obtained from the Kubernetes API; further includes:

[0105] Use metadata.labels to add monitoring labels in the Pod's YAML definition;

[0106] Use metadata.annotations to add monitoring-related metadata to the Pod's YAML definition;

[0107] In the Prometheus configuration file, use the relabel_configs configuration item to filter, rename, add, and modify labels obtained from the Kubernetes API.

[0108] The Shell-Operator integration module is used to monitor Kubernetes events and dynamically set or modify the monitoring labels of Pods through shell scripts based on event types and Pod attributes; further includes:

[0109] Write a shell script to monitor events of Pods in a specified namespace in Kubernetes;

[0110] Dynamically set or modify monitoring labels and annotations for Pods based on their current labels, node information, and event types;

[0111] Identify the services that need to be monitored by checking the labels and annotations of the Pods, and add necessary monitoring labels to them so that Prometheus can automatically discover and monitor them.

[0112] The Consul integration module allows users to set custom monitoring labels for services in Consul through the interface or API. Prometheus performs service discovery and monitoring based on these labels; further includes:

[0113] Allow users to set custom monitoring tags for services in Consul through the interface or API;

[0114] Tags are stored as part of the service registration information and queried through the Consul API;

[0115] Prometheus uses Consul as a service discovery source and filters and selects services to be monitored based on labels.

[0116] Helm Charts integration module, which allows setting monitoring-related labels and annotations for Kubernetes resources in Charts YAML definitions;

[0117] The Exporter configuration module is used to specify monitoring indicators in the Exporter configuration and add custom labels so that the monitoring data with labels can be sent to the Prometheus server; further includes:

[0118] Specify the monitoring indicators to be collected in the Exporter configuration and add custom labels to them;

[0119] Tags are sent to the Prometheus server as part of the monitoring data, enabling refined classification and management of monitoring data;

[0120] Different Exporters collect monitoring data for different applications or services and add unique labels to the data for query and analysis in Prometheus.

[0121] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for dynamically setting labels for a monitoring system based on Prometheus, characterized in that: The method comprises the following steps: In the Prometheus configuration file, use the relabel_configs configuration item to filter, rename, and add labels obtained from the Kubernetes API to dynamically set monitoring labels for services and pods in Kubernetes. Use the Shell-Operator tool to write and execute shell scripts, monitor Kubernetes events, and dynamically set or modify monitoring labels for Pods based on Pod labels and node information, ensuring that Prometheus effectively collects and classifies monitoring data based on labels. In the Consul service registration and discovery system, users are allowed to set custom monitoring labels for services through the interface or API, and Prometheus automatically discovers and monitors the corresponding services from Consul based on the labels; In the YAML definition of Helm Charts, set monitoring-related labels and annotations for Kubernetes resources so that Prometheus can identify and collect monitoring data for resources; In the Exporter configuration, specify the monitoring indicators to be collected and add custom labels to them. The monitoring data with labels is sent to the Prometheus server through Exporter to achieve refined classification and management of monitoring data.

2. According to a prometheus-based monitoring system dynamic label setting method according to claim 1, it is characterized in that: The relabel_configs configuration item is used in the Prometheus configuration file to modify the labels obtained from the Kubernetes API, including but not limited to filtering unnecessary labels, renaming labels, and adding new monitoring labels.

3. According to a prometheus-based monitoring system dynamic label setting method according to claim 2, it is characterized in that: The Shell-Operator tool monitors Kubernetes Pod creation, update or deletion events, combines the Pod's labels and node information, executes a custom shell script, dynamically sets or modifies annotations and labels for the Pod, and then implements fine-grained monitoring of the Pod in Prometheus.

4. According to a prometheus-based monitoring system dynamic label setting method according to claim 1, it is characterized in that: In the Consul service registration and discovery system, users set custom monitoring tags for services through the interface or API. The tags are part of the service registration information and are read by Prometheus during service discovery and applied to the selection of monitoring targets and data collection.

5. According to a prometheus-based monitoring system dynamic label setting method according to claim 1, it is characterized in that: When defining Kubernetes resources, the Helm Charts add monitoring-related configurations to the metadata.labels and metadata.annotations of the resources, so that Prometheus automatically discovers and collects monitoring data of the resources based on the configuration, thereby realizing dynamic label setting and classification management of the monitoring data.

6. A prometheus-based monitoring system dynamic label setting method according to any one of claims 1 to 5, characterized in that: The system comprises: A Prometheus monitoring system to collect and process monitoring data; Configuration module for setting the external_labels property in Prometheus so that these labels are automatically added to all metrics when querying or exporting data; Job configuration module, which is used to divide different Prometheus jobs according to the type of monitoring target, so as to achieve refined division of monitoring data; Kubernetes integration module, which automatically discovers services and pods in the cluster through Kubernetes' Service Discovery mechanism and relabel_configs configuration item, and allows filtering, renaming, adding and modifying labels obtained from the Kubernetes API; Shell-Operator integration module, which monitors Kubernetes events and dynamically sets or modifies Pod monitoring labels through shell scripts based on event types and Pod attributes; Consul integration module, which allows users to set custom monitoring labels for services in Consul through the interface or API. Prometheus performs service discovery and monitoring based on these labels; Helm Charts integration module, which allows setting monitoring-related labels and annotations for Kubernetes resources in Charts YAML definitions; The Exporter configuration module is used to specify monitoring indicators in the Exporter configuration and add custom labels so that the monitoring data with labels can be sent to the Prometheus server.

7. According to claim 6, a prometheus-based monitoring system dynamic label setting system is characterized by: The Kubernetes integration module further includes: Use metadata.labels to add monitoring labels in the Pod's YAML definition; Use metadata.annotations to add monitoring-related metadata to the Pod's YAML definition; In the Prometheus configuration file, use the relabel_configs configuration item to filter, rename, add, and modify labels obtained from the Kubernetes API.

8. According to claim 6, a prometheus-based monitoring system dynamic label setting system is characterized by: The Shell-Operator integration module further includes: Write a shell script to monitor events of Pods in a specified namespace in Kubernetes; Dynamically set or modify monitoring labels and annotations for Pods based on their current labels, node information, and event types; Identify the services that need to be monitored by checking the labels and annotations of the Pods, and add necessary monitoring labels to them so that Prometheus can automatically discover and monitor them.

9. According to claim 6, a prometheus-based monitoring system dynamic label setting system is characterized by: The Consul integration module further includes: Allow users to set custom monitoring tags for services in Consul through the interface or API; Tags are stored as part of the service registration information and queried through the Consul API; Prometheus uses Consul as a service discovery source and filters and selects services to be monitored based on labels.

10. According to claim 6, a prometheus-based monitoring system dynamic label setting system is characterized by: The Exporter configuration module further includes: Specify the monitoring indicators to be collected in the Exporter configuration and add custom labels to them; Tags are sent to the Prometheus server as part of the monitoring data, enabling refined classification and management of monitoring data; Different Exporters collect monitoring data for different applications or services and add unique labels to the data for query and analysis in Prometheus.

Images