A PROMETHEUS dynamic alarm rule management method based on CONFD

Abstract

The present invention discloses a PROMETHEUS dynamic alarm rule management method implemented based on CONFD, by obtaining dynamic alarm rules from centralized unified configuration management center confd, and automatically assembling them into the alarm file format of prometheus standard; and the source of the alarm rule is to realize configuration by the custom module written in JAVA, and the custom module is used as the back-end system of confd, and confd can automatically query and dynamically monitor the custom module in real time. After detecting that the configuration has changed, the prometheus alarm rule file will be re-rendered as the latest version on the custom module, and the alarm rule is made to take effect immediately by hot loading instructions. The present invention realizes the dynamic configuration of alarm rules under the premise of no code intrusion, is safe, reliable and has strong scalability, and greatly improves operation and maintenance efficiency.

Description

Technical Field

[0001] The present invention relates to a PROMETHEUS dynamic alarm rule management based on CONFD, belonging to the technical field of system monitoring and operation and maintenance. Background Art

[0002] With the rapid development of internet services, users are placing increasingly high demands on their systems. Effective monitoring safeguards systems and effectively improves their reliability, availability, and user experience. Monitoring systems are crucial for the entire operations and maintenance process, and indeed, the entire project and product lifecycle. Leveraging its unique advantages in cloud-native and containerized monitoring, Prometheus is becoming a leading next-generation monitoring solution.

[0003] The core function of real-time monitoring is to detect problems and notify users immediately. Like most systems, Prometheus's alarm rules are configured through configuration files. Each time an alarm rule is added or modified, the configuration file must be manually modified and the Prometheus service restarted for the new configuration to take effect. This is undoubtedly inefficient and hinders developers from customizing rules. All configuration relies on Prometheus operations and maintenance personnel, which increases communication costs.

[0004] Traditional configuration management systems are code-invasive, requiring applications to use SDKs to query and monitor data changes. For some mature systems, integrating SDKs to implement dynamic configuration management is difficult, especially for a large project like Prometheus.

[0005] How to quickly and efficiently implement Prometheus alarm rule configuration is a technical problem that technical personnel in this field urgently need to solve. Summary of the Invention

[0006] Objective: To overcome the shortcomings of the prior art, the present invention provides a PROMETHEUS dynamic alarm rule management method based on CONFD. The confd service is a lightweight configuration management tool that can implement dynamic configuration management of third-party systems by querying the backend storage system, thereby changing the system configuration without code intrusion.

[0007] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A prometheus dynamic alarm rule management method based on confd includes the following steps:

[0009] Add and publish prometheus rules from a custom JAVA module.

[0010] Configure the core parameters of the confd service, including template files, dynamic target rendering file paths, rule key sets, and Prometheus hot loading instructions.

[0011] Start the confd service and set the custom JAVA module as the backend storage of the confd service in the startup command line.

[0012] When prometheus rules are added, modified, or deleted in the front-end UI page corresponding to the custom JAVA module, the confd service perceives the change of prometheus rules through the backend storage, renders the changed prometheus rules grouped and saved in the rule key set into the template file, and writes the rendered template file to the dynamic target rendering file path.

[0013] Prometheus hot loading instructions execute the template file under the dynamic target rendering file path.

[0014] As a preferred solution, the process of adding and publishing prometheus rules from a custom JAVA module includes the following steps:

[0015] Add prometheus rules from the page and fill in the unique identification key and description of the current rules.

[0016] Enter the specific content of rules through key-value pairs.

[0017] Verify the specific content of the entered rules.

[0018] The specific content of the rules is stored in a JSON array named with the unique identifier key in the background of the custom JAVA module.

[0019] As a preferred solution, the unique identification key corresponds to the rule key set in the confd service. A unique identification key can correspond to multiple rules, which is equivalent to the group name of rules of the same type.

[0020] As a preferred solution, the configuration of the confd service core parameters includes the following steps:

[0021] Create a template file in the template folder of the confd service installation path / etc / confd. Attribute values ​​in the template file are represented by placeholders "{{}}". Attribute values ​​come from the Prometheus rules of the custom Java module.

[0022] Create the core configuration file prometheus_rules.toml for the confd service. The default directory of this file is the / etc / confd / conf.d folder. Configure the four core parameters src, dest, keys and reload_cmd. src represents the template file, dest represents the dynamic target rendering file path, keys represents the rule key set, and reload_cmd represents the Prometheus hot reload instruction.

[0023] As a preferred solution, starting the confd service and setting the custom JAVA module as the backend storage of the confd service in the startup command line includes the following steps:

[0024] In the command line parameters for starting the confd service, the backend parameter is set to the name of the backend microservice corresponding to the custom JAVA module, the node parameter is set to the backend service address of the custom JAVA module, and the watch parameter is set to enable the confd service to support dynamic monitoring.

[0025] As a preferred solution, the confd service is also customized as follows:

[0026] Add the java-client type to the backend type in the confd service.

[0027] When the confd service is started, the command-line node parameter specifies the TCP service port of the custom Java module, establishing a persistent TCP connection. A persistent TCP connection message processing module is added to the confd service. When the confd service receives a rules change message, it invokes its own logic to dynamically render the configured template file.

[0028] Modify the command line parameters for starting the confd service. When the backend parameter in the startup command line is set to the java-client type service, the logical processing mode switches from the confd service's own mode to TCP long connection listening, and the TCP address and port of the connection are converted to the node parameter in the startup command line.

[0029] Beneficial effects: The present invention provides a PROMETHEUS dynamic alarm rule management method based on CONFD, which obtains dynamic alarm rules from the centralized unified configuration management center confd and automatically assembles them into the prometheus standard alarm file format; the source of the alarm rules is configured through a custom module written in JAVA, and the custom module is used as the back-end system of confd. confd can automatically query and dynamically monitor the custom module in real time. After detecting that the configuration has changed, it will re-render the prometheus alarm rule file as the latest version on the custom module, and make the alarm rule take effect immediately through hot loading instructions. This method realizes the dynamic configuration of alarm rules without code intrusion, is safe, reliable and highly scalable, and greatly improves operation and maintenance efficiency. Its advantages are as follows:

[0030] 1. No code intrusion, reducing development complexity.

[0031] 2. Implemented dynamic configuration of alarm rules without manually restarting the Prometheus service.

[0032] 3. Easy to use, flexible configuration and convenient deployment.

[0033] 4. Provide WEB UI for developers and operation and maintenance personnel to use, which is more convenient and reduces communication costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below with reference to specific embodiments.

[0036] When the method of the present invention is applied, three software are involved, which are as follows:

[0037] 1. Prometheus service: monitoring service for collecting program indicators and processing alarms.

[0038] 2. confd service: dynamic configuration management service.

[0039] 3. Custom JAVA module: Java microservice, used for users to enter custom alarm rules.

[0040] A prometheus dynamic alarm rule management method based on confd includes the following steps:

[0041] Add and publish prometheus rules from a custom JAVA module.

[0042] Configure the core parameters of the confd service, including template files, dynamic target rendering file paths, rule key sets, and Prometheus hot loading instructions.

[0043] Start the confd service and set the custom JAVA module as the backend storage of the confd service in the startup command line.

[0044] When prometheus rules are added, modified, or deleted in the front-end UI page corresponding to the custom JAVA module, the confd service perceives the change of prometheus rules through the backend storage, renders the changed prometheus rules grouped and saved in the rule key set into the template file, and writes the rendered template file to the dynamic target rendering file path.

[0045] The Prometheus hot loading instruction executes the template file under the dynamic target rendering file path, and the changed Prometheus rules take effect.

[0046] Preferably, adding and publishing prometheus rules from a custom JAVA module includes the following steps:

[0047] Add prometheus rules from the page and fill in the unique identification key and description of the current rules.

[0048] Enter the specific content of rules through key-value pairs.

[0049] Verify the specific content of the entered rules to prevent invalid alarm rules from being generated.

[0050] The specific content of the rules is stored in a JSON array named with the unique identifier key in the background of the custom JAVA module.

[0051] The unique identifier key is required. The unique identifier key corresponds to the rule key set in the confd service. A unique identifier key can correspond to multiple rules, which is equivalent to the group name of rules of the same type.

[0052] Preferably, configuring the confd service core parameters includes the following steps:

[0053] Create a template file in the template folder of the confd service installation path / etc / confd. Attribute values ​​in the template file are represented by placeholders "{{}}". Attribute values ​​come from the Prometheus rules of the custom Java module.

[0054] Create the core configuration file prometheus_rules.toml for the confd service. The default directory of this file is the / etc / confd / conf.d folder. Configure the four core parameters src, dest, keys and reload_cmd. src represents the template file, dest represents the dynamic target rendering file path, that is, the alarm rule storage path configured in the prometheus service, keys represents the rule key set, and reload_cmd represents the prometheus hot reload instruction.

[0055] Preferably, starting the confd service and setting the custom JAVA module as the backend storage of the confd service in the startup command line includes the following steps:

[0056] In the command line parameters for starting the confd service, the backend parameter is set to the name of the backend microservice corresponding to the custom JAVA module, the node parameter is set to the backend service address of the custom JAVA module, and the watch parameter is set to enable the confd service to support dynamic monitoring.

[0057] Preferably, the confd service can dynamically read the prometheus rules stored in the JAVA custom module by customizing the confd service source code as follows:

[0058] 1. Add the java-client type to the backend type in the confd service.

[0059] 2. Write the monitoring logic for the custom Java module. In this example, the custom Java module has two core purposes: first, it stores the Prometheus rules entered by users, as mentioned above, and second, it acts as a TCP server to connect to clients that need to communicate with it (in this example, the client is the confd service). When the custom Java module starts, it opens a port for external TCP connections. When the confd service is started, the command-line node parameter specifies this TCP service port, establishing a persistent TCP connection between the confd service and the custom Java module. When the Prometheus rules stored in the custom Java module change, the custom Java module sends a change message to this connection. We need to add a TCP persistent connection message processing module to the confd service. Upon receiving the rule change message, the confd service invokes its own logic to dynamically render the configured template file.

[0060] 3. Modify the command line parameters for the confd service startup. When the backend parameter in the startup command line is set to a java-client service type, the logical processing mode is automatically switched from the confd service's native mode to TCP persistent connection listening. The TCP address and port of the connection are converted to the node parameter in the startup command line. This allows custom Java modules to be used as the backend storage unit of the confd service.

[0061] Add / modify / delete prometheus rules on the front-end UI page corresponding to the custom JAVA module, turn the custom JAVA module into the storage backend of the confd service, and dynamically update the local configuration file in combination with the configuration template engine to keep the data consistent with the backend.

[0062] Example 1:

[0063] like Figure 1 As shown, a PROMETHEUS dynamic alarm rule management method based on CONFD includes the following steps:

[0064] Step 1: Start the promtheus service and set the path to the alarm rule file in its core configuration file prometheus.yml.

[0065] Step 2: Write the core logic code for the confd service backend storage.

[0066] Step 3: Create the confd service configuration, including the template file, dynamic target rendering file path, rule key set, and Prometheus hot loading instructions.

[0067] Step 4: Customize the confd service source code to support a custom JAVA module as its backend storage.

[0068] Step 5: On the front-end UI page corresponding to the custom JAVA module, add alarm rule configuration content and key.

[0069] Step 6: Start the confd service and set the above configuration to the startup related command line parameters.

[0070] Step 7: Log in to the Prometheus UI and you can see that the configured alarm rules have taken effect. You can also modify the alarm rule configuration on the WEB UI page at any time, and the new rules will take effect immediately.

[0071] Example 2:

[0072] The present invention innovatively introduces the confd service as a configuration management tool for proemtehus. The backend storage (backend) currently supported by the confd service includes etcd, zookpper, nacos, consul, etc. In this example, dynamic configuration can also be achieved by combining the above backends, but the scalability of later functions is greatly reduced. Therefore, the present invention adopts the method of connecting a custom backend storage (backend) to the confd service. Although there is a lot of code development, there is a lot of room and flexibility for the expansion and customization of the custom backend in the future.

[0073] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A PROMETHEUS dynamic alarm rule management method based on CONFD, characterized by: The steps include: Add and publish prometheus rules from a custom JAVA module; Configure the core parameters of the confd service, including template files, dynamic target rendering file paths, rule key sets, and Prometheus hot loading instructions; Start the confd service and set the custom JAVA module as the backend storage of the confd service in the startup command line; When adding / modifying / deleting Prometheus rules in the front-end UI page corresponding to the custom JAVA module, the confd service perceives the change of Prometheus rules through the backend storage, renders the changed Prometheus rules grouped and saved in the rule key set into the template file, and writes the rendered template file to the dynamic target rendering file path; Prometheus hot loading instructions execute the template file under the dynamic target rendering file path; Starting the confd service and setting the custom JAVA module as the backend storage of the confd service in the startup command line include the following steps: In the command line parameters for starting the confd service, the backend parameter is set to the name of the backend microservice corresponding to the custom JAVA module, the node parameter is set to the backend service address of the custom JAVA module, and the watch parameter is set to enable the confd service to support dynamic monitoring; The confd service is also customized as follows: Add java-client type to the backend type of confd service; When the confd service is started, the command line node parameter specifies the TCP service port of the custom Java module, establishing a TCP persistent connection. A TCP persistent connection message processing module is added to the confd service. When the confd service receives a rules change message, it calls the confd service's own logic to dynamically render the configured template file. Modify the command line parameters for starting the confd service. When the backend parameter in the startup command line is set to the java-client type service, the logical processing mode switches from the confd service's own mode to TCP long connection listening, and the TCP address and port of the connection are converted to the node parameter in the startup command line.

2. A PROMETHEUS dynamic alarm rule management method based on CONFD according to claim 1, characterized in that: Adding and publishing prometheus rules from a custom JAVA module includes the following steps: Add prometheus rules from the page and fill in the unique identification key and description of the current rules; Enter the specific content of rules through key-value pairs; Verify the specific content of the entered rules; The specific content of the rules is stored in a JSON array named with the unique identifier key in the background of the custom JAVA module.

3. A CONFD-based PROMETHEUS dynamic alarm rule management method according to claim 2, characterized in that: The unique identification key corresponds to the rule key set in the confd service. A unique identification key can correspond to multiple rules, which is equivalent to the group name of rules of the same type.

4. The PROMETHEUS dynamic alarm rule management method based on CONFD according to claim 1 is characterized in that: Configuring the core parameters of the confd service includes the following steps: Create a template file in the template folder of the confd service installation path / etc / confd. The attribute values ​​in the template file are represented by placeholders "{{}}". The attribute values ​​come from the prometheus rules of the custom JAVA module. Create the core configuration file prometheus_rules.toml for the confd service. The default directory of this file is the / etc / confd / conf.d folder. Configure the four core parameters src, dest, keys and reload_cmd. src represents the template file, dest represents the dynamic target rendering file path, keys represents the rule key set, and reload_cmd represents the Prometheus hot reload instruction.

Images