A method and system for persisting and synchronizing Prometheus configuration based on containerization

Abstract

The application provides a container-based Prometheus configuration persistence and real-time synchronization method and system, relates to the cloud native monitoring technical field, and comprises the following steps: stripping Prometheus monitoring target configuration information from a Prometheus configuration file and persistently storing the Prometheus monitoring target configuration information in an external database to obtain persistent monitoring target configuration information; developing and deploying a custom Consul service component, calling the persistent monitoring target configuration information through the custom Consul service component, and constructing a simulated API interface; establishing a configuration information synchronization mechanism for the custom Consul service component and the external database; and realizing dynamic management of the monitoring target configuration information by Prometheus based on the custom Consul service component, the simulated API interface and the configuration information synchronization mechanism. The application eliminates image reconstruction, realizes configuration persistence, supports dynamic real-time updating, is lightweight and has strong universality, and effectively improves operation and maintenance efficiency and reliability.

Description

Technical Field

[0001] This invention relates to the field of cloud-native monitoring technology, and in particular to a containerized Prometheus configuration persistence and real-time synchronization method and system. Background Technology

[0002] Prometheus, as a widely adopted open-source monitoring and alerting tool, plays a crucial role in cloud-native environments. Its core configuration, especially the scrape_configs section that defines monitoring targets, is typically stored in a configuration file named prometheus.yml.

[0003] In traditional virtual machine deployments, modifying the monitoring target typically only requires editing this configuration file and restarting the Prometheus service. However, in modern containerized deployment environments (such as Docker and Kubernetes), Prometheus is usually packaged as a container image. This presents significant configuration management challenges: 1. Non-persistent configuration files: Container file systems are typically temporary. Modifications to the prometheus.yml file within the container will be lost when the container restarts or is rebuilt. While persistence can be achieved by mounting host files onto Docker volumes, this increases the complexity of host management and makes synchronization difficult in multi-node clusters.

[0004] 2. High Image Coupling: Packaging the prometheus.yml file into a Docker image is a common practice. However, this means that any modification to the monitored target (such as adding a Job or Target) requires rebuilding the PrometheusDocker image and redeploying the container. This process is cumbersome and time-consuming, violates the best practice of containers being "immutable infrastructure," and is extremely inefficient in scenarios where monitoring targets need to be frequently adjusted.

[0005] 3. Limitations of dynamic discovery solutions: Prometheus provides various service discovery mechanisms (such as consul_sd_configs, kubernetes_sd_configs, file_sd_configs) to achieve dynamic management of monitoring targets.

[0006] (1) Consul Service Discovery: Although `consul_sd_configs` can achieve dynamic target discovery, it relies on an external Consul cluster. Consul itself also needs to perform service registration (usually done by the monitored application or registry) and configuration management. Introducing Consul increases system complexity, operational burden, and potential points of failure. More importantly, Consul's own service registration information also needs to be persistently stored, essentially not completely eliminating the need for persistent management, but merely shifting the problem.

[0007] (2). Kubernetes service discovery: kubernetes_sd_configs is only applicable to Kubernetes environments and mainly discovers resources such as Kubernetes Services and Pods. It has limited capabilities for non-Kubernetes managed targets or more complex crawling and configuration management.

[0008] (3) File service discovery: file_sd_configs allows the discovery of targets by reading external JSON or YAML files, avoiding the need to modify the main configuration file. However, this requires solving the problems of file persistence, dynamic updates (usually requiring external processes or APIs), and shared storage in container environments (such as using shared volumes), making management still not convenient or direct enough.

[0009] How to solve the above-mentioned technical problems is the challenge facing this invention. Summary of the Invention

[0010] To address the shortcomings of existing technologies, this invention provides a containerized Prometheus configuration persistence and real-time synchronization method and system that eliminates image reconstruction, achieves configuration persistence, supports dynamic real-time updates, is lightweight and highly versatile, and effectively improves operational efficiency and reliability.

[0011] The technical solution adopted by this invention to solve its technical problem is as follows: This invention provides a containerized Prometheus configuration persistence and real-time synchronization method, including the following steps: Extract the Prometheus monitoring target configuration information from the Prometheus configuration file and persist it to an external database to obtain persistent monitoring target configuration information; the persistent monitoring target configuration information includes at least the monitoring task name and the address list of the monitored targets; Develop and deploy a custom Consul service component, and use the custom Consul service component to call persistent monitoring target configuration information to build a simulated API interface; the custom Consul service component is used to simulate the HTTP API interface of the Consul service. Establish a configuration information synchronization mechanism between custom Consul service components and external databases; Prometheus achieves dynamic management of monitoring target configuration information by using a custom Consul service component, a simulated API interface, and a configuration information synchronization mechanism.

[0012] Preferably, the persistent monitoring target configuration information also includes the capture interval, timeout period, indicator path, protocol type, and tag.

[0013] Preferably, the custom Consul service component is deployed in a containerized manner.

[0014] Preferably, the simulated API interface includes GET / v1 / agent / services: Returns a list of all monitoring tasks and monitored targets' addresses that are persistently stored in an external database; GET / v1 / health / service / {service_name}: Returns the health status data of the monitored target based on the monitoring task name; GET / v1 / agent / self: Returns the preset Agent metadata; GET / v1 / catalog / services: Returns the mapping relationship between all monitoring tasks and their tags; PUT / v1 / agent / service / register: Receives external registration requests and writes the data to an external database.

[0015] Preferably, the establishment of the configuration information synchronization mechanism adopts any one of the following: listening to change notifications from external databases, periodically polling persistently monitoring target configuration information, parsing binary logs or change data streams from external databases.

[0016] Preferably, the dynamic management of the monitoring target configuration information by Prometheus based on a custom Consul service component, a simulated API interface, and a configuration information synchronization mechanism includes setting the consul_sd_configs service discovery mode in the Prometheus configuration file and pointing the service address to the custom Consul service component. Prometheus calls the simulated API interface and combines it with the configuration information synchronization mechanism to achieve dynamic management of the monitoring target configuration information.

[0017] Preferably, configuring the consul_sd_configs service discovery method includes configuring the network address and port of a custom service, optional monitoring task filtering parameters, and optional tag filtering parameters.

[0018] Preferably, the external database includes a relational database and a key-value database.

[0019] This invention also provides a containerized Prometheus configuration persistence and real-time synchronization system, including... The configuration storage module is used to extract the Prometheus monitoring target configuration information from the Prometheus configuration file, persist the monitoring target configuration information to an external database, and output the persistent monitoring target configuration information. A custom service module is used to develop and deploy custom Consul service components in a containerized manner, and to call the persistent monitoring target configuration information output by the configuration stripping and persistence module to build a simulated API interface; The configuration information synchronization module is used to establish a configuration information synchronization mechanism between the custom Consul service component module and the external database; The dynamic discovery configuration module is used to set the service discovery method of consul_sd_configs, point the service address of the service discovery method to the custom Consul service component module, and trigger Prometheus to call the simulated API interface. Combined with the configuration information synchronization mechanism, it enables Prometheus to dynamically manage the configuration information of the monitoring target.

[0020] The beneficial effects of this invention are: eliminating image rebuilding, achieving configuration persistence, supporting dynamic real-time updates, lightweight and highly versatile, effectively improving operational efficiency and reliability. Any add, delete, or modify operation on the monitored target only requires updating the external database record. The configuration of the Prometheus container itself (prometheus.yml) is completely fixed (containing only the consul_sd_configs configuration pointing to the custom service), thus avoiding the need to rebuild the Prometheus Docker image due to monitoring configuration changes, greatly simplifying the operation and maintenance process of containerized Prometheus. The monitoring target configuration is stored in an external database, independent of the lifecycle of the Prometheus container instance. Container restarts, destruction, or scheduling to a new node can automatically restore the correct monitoring configuration through the database and custom service, ensuring configuration security and reliability. Prometheus periodically pulls the configuration through its service discovery mechanism, thereby achieving dynamic, near real-time updates of the monitored target without manual intervention or restarting Prometheus. Through a few key APIs, a custom lightweight service avoids the resource consumption, operational complexity, and additional dependencies associated with deploying and maintaining a complete Consul cluster. It is not dependent on a specific container orchestration platform (such as Kubernetes) and is applicable to any Prometheus environment deployed based on the Docker container technology stack, demonstrating excellent versatility. Operations personnel can manage all monitoring targets by manipulating the database (using standard SQL, a management UI, or an API), making the operation simple, centralized, and auditable. It significantly improves the efficiency and accuracy of configuration management in large-scale monitoring environments and reduces the risk of human error. Attached Figure Description

[0021] Figure 1 This is a diagram illustrating the method steps of the present invention.

[0022] Figure 2 This is a system module diagram of the present invention.

[0023] Figure 3 This is a diagram illustrating the system architecture and component interactions of the present invention.

[0024] Figure 4 This is a flowchart of the real-time synchronization mechanism for configuration information of the present invention.

[0025] Figure 5 This is an internal logic diagram of the custom Consul service component of this invention. Detailed Implementation

[0026] To clearly illustrate the technical features of this solution, the following detailed implementation method will be used to explain the solution.

[0027] Example 1: See Figure 1 , 3 As shown in Figures 4 and 5, this embodiment is a containerized Prometheus configuration persistence and real-time synchronization method, which includes the following steps: S1. Extract the Prometheus monitoring target configuration information from the Prometheus configuration file and persist it to an external database to obtain persistent monitoring target configuration information; the persistent monitoring target configuration information includes at least the monitoring task name and the list of addresses of the monitored targets; External databases include relational databases and key-value databases.

[0028] The persistent monitoring target configuration information also includes the capture interval, timeout, indicator path, protocol type, and tags.

[0029] S2. Develop and deploy a custom Consul service component. Use the custom Consul service component to call persistent monitoring target configuration information and build a simulated API interface. The custom Consul service component is used to simulate the HTTP API interface of the Consul service, but its internal logic does not depend on Consul itself. Deploy custom Consul service components in a containerized manner.

[0030] Simulated API interface includes GET / v1 / agent / services: Returns a list of all monitoring tasks and monitored targets' addresses that are persistently stored in an external database; GET / v1 / health / service / {service_name}: Returns the health status data of the monitored target based on the monitoring task name; GET / v1 / agent / self: Returns the preset Agent metadata; GET / v1 / catalog / services: Returns the mapping relationship between all monitoring tasks and their tags; PUT / v1 / agent / service / register: Receives external registration requests and writes the data to an external database.

[0031] It should be noted that GET / v1 / agent / services returns all services registered by the current "Agent" (i.e., this custom service). In implementation, the service component queries the database described in S1 to obtain all job_names and their corresponding targets, treating each job_name as a Consul "Service" and each target as a "Service Instance" of that service. It constructs and returns a JSON response conforming to the Consul API format (containing information such as service ID, service name, address, port, and tags).

[0032] GET / v1 / agent / self: Returns the configuration and member information of the current "Agent" (i.e., this custom service). In implementation, the service component returns preset fixed values ​​or simply generated metadata (such as Datacenter in Config) to meet the API call requirements of Prometheus.

[0033] GET / v1 / health / service / {serviceName}: Returns the health status information of the specified service name (serviceName). This is the core interface for Prometheus service discovery. In its implementation, the service component maps the passed-in serviceName to a job_name in the database and queries the list of targets corresponding to that job_name. For each target, its information (address, port) and status (usually returning a passing status) are constructed into a JSON format conforming to the Consul health check response and returned. Prometheus primarily uses the list of addresses returned by this interface to determine crawling targets.

[0034] GET / v1 / catalog / services: Returns all services and their associated tags. In implementation, the service component queries the database to obtain a list of all job_names and optionally appends some tag information (such as reading from the database configuration or pre-set), constructing a JSON response conforming to the Consul API format (a mapping of service names to a list of tags).

[0035] PUT / v1 / agent / service / register: Used to register a service. Although Prometheus itself does not call this interface to register targets (target registration is done by the monitored endpoint or the registrar), implementing this interface can be used to simulate registration or for external tools to call. During implementation, the service component parses the request body (containing information such as service name, address, port, and tags) and writes or updates this information to the database described in S1.

[0036] S3. Establish a configuration information synchronization mechanism between custom Consul service components and external databases; The configuration information synchronization mechanism can be established by listening to change notifications from external databases, periodically polling persistently monitored target configuration information, parsing binary logs or change data streams from external databases, or by using any one of these methods.

[0037] It should be noted that database listeners / triggers: If the database supports this (such as LISTEN / NOTIFY in PostgreSQL), configure the database to notify a custom service when the data in the relevant tables changes.

[0038] Active polling: The custom service periodically queries the relevant tables in the database to check for any new, modified, or deleted records, and updates its internal status accordingly.

[0039] Change log tracking: Read the database's Binlog or Change Data Capture (CDC) stream to detect data changes in real time.

[0040] When a change in the monitoring target configuration is detected in the database, the custom service immediately updates its internally maintained service and instance information cache to ensure that subsequent API requests return the latest data.

[0041] S4. Based on a custom Consul service component, simulated API interface, and configuration information synchronization mechanism, Prometheus achieves dynamic management of the configuration information of the monitored target.

[0042] Configure the consul_sd_configs service discovery method and point the service address to the custom Consul service component. Prometheus calls the simulated API interface and combines the configuration information synchronization mechanism to achieve dynamic management of the configuration information of the monitoring target.

[0043] Setting up service discovery in consul_sd_configs includes configuring the network address and port of custom services, monitoring task filtering parameters, and tag filtering parameters.

[0044] It should be noted that the monitoring task filtering parameters are used to filter specific services (Jobs). If not specified or empty, all services (corresponding to all Jobs) on this Consul Agent (i.e., the custom service) will be discovered. The tag filtering parameters are used to filter service instances based on tags.

[0045] Other crawling-related configurations (such as scrape_interval, metrics_path, etc.) can be defined here or at the Job level.

[0046] According to the Prometheus configuration cycle (usually scrape_interval), Prometheus will automatically call the aforementioned API (mainly / v1 / health / service / {serviceName}) of the configured custom service address to obtain the latest list of monitoring targets and start scraping. This enables dynamic loading of the monitoring target configuration.

[0047] Example 2: To verify the effectiveness of this invention, this embodiment uses a lightweight implementation based on Java and MySQL to conduct a comparative test on the effect of the method proposed in this invention. The specific implementation steps include: S1: Extract the Prometheus monitoring target configuration information from the Prometheus configuration file and persist it to an external database to obtain persistent monitoring target configuration information; Database selection and table design MySQL was chosen as the external database. The prometheus_configs table was created to store the monitoring target configurations. The table structure is as follows: CREATE TABLE prometheus_configs ( job_name VARCHAR(50) PRIMARY KEY, targets JSON NOT NULL COMMENT 'List of monitored target addresses, such as ["host1:9090","host2:9090"]', scrape_interval INT DEFAULT 30 COMMENT 'Fetch interval (seconds)', metrics_path VARCHAR(100) DEFAULT ' / metrics' COMMENT 'metric path', labels JSON COMMENT 'Custom labels, such as {"env":"production"}' ); Initial data example: INSERT INTO prometheus_configs VALUES ('node_exporter', '["node01:9100", "node02:9100"]', 30, ' / metrics', '{"env":"prod"}'); S2: Develop and deploy a custom Consul service component, and use the custom Consul service component to call persistent monitoring target configuration information and build a simulated API interface; Develop independent service components using the Java language and deploy them in containers (image size <20MB).

[0048] Functionality: Based on the Spring Boot framework, this is a monolithic service with two core functionalities. First, it monitors database configuration changes in real time. When the service initializes or the database configuration changes, it simulates a Consul HTTP API interface to provide the monitored data to the Prometheus service.

[0049] Startup command: java -Xmx2048m -Xms2048m -jar custom-consul-service.jar --server.port=$PORT --spring.config.location=classpath: / bootstrap-prod.ym GET / v1 / agent / services interface (1). Function: Returns all registered monitoring jobs and their basic information. (2). Implementation logic: (2.1). Query the database to retrieve all job configurations (job_name and corresponding target list). (2.2). Convert each job_name to a Consul service object: Service ID format: job-{job_name} Use job_name directly for the service name. Convert the labels field in the database to Consul label format. (2.3). Leave the service address and port blank (the specific instance address will be returned by the health interface). (3). Output data format: { "node_exporter": { "ID": "job-node_exporter", "Service": "node_exporter", "Tags": ["env=prod"], "Address": "", "Port": 0 } } GET / v1 / health / service / {service_name} interface (1). Function: Returns the health status of the specified Job and details of the monitoring target (Prometheus core discovery interface) (2). Implementation logic: (2.1). Extract service_name (corresponding to job_name) from the URL path parameters. (2.2). Query the database to obtain the list of targets for this job. (2.3). Resolve each target address: Split IP / Hostname and port Generate a unique instance ID: {service_name}-target-{serial number} (2.4). Construct a health check status for each instance (always returns "passing"). (2.5). Assemble the service instance list in Consul format. (3). Output data format: [ { "Service": { "ID": "redis-target-0", "Service": "redis", Address: 10.0.0.1 "Port": 6379 }, "Checks": [{"Status": "passing"}] } ] S3.3. GET / v1 / agent / self interface Function: Returns the metadata of the service node. Implementation logic: (1). Return to the default fixed configuration: The Datacenter field is set to "dc1" Set the node name to "node1" The IP address used is the actual IP assigned by the container runtime. (2) Other fields should be left empty or have default values. Output data format: { "Config": {"Datacenter": "dc1"}, "Member": { "Name": "node1", "Addr": "192.168.10.1" } } GET / v1 / catalog / services interface Function: Returns the tag mapping relationship of all Jobs. Implementation logic: (1) Query the database to retrieve all job_names and labels. (2). Convert the labels dictionary into an array of label strings. (3) Construct a mapping dictionary from service names to the tag list. Output data format: { "service1": ["v1"], "service2": ["v2"] } PUT / v1 / agent / service / register interface Function: Receives external registration requests and writes them to the database. Implementation logic: (1) Parse the service name, address, port, and tag in the JSON request body. (2) Convert the label array to key-value format (3). Construct the target address: {Address}:{Port} (4). Write to the database: If job_name exists, update the targets list. If the job record does not exist, create a new job record. Output data format: { "Name": "redis_exporter", Address: 10.0.0.1 "Port": 9121, "Tags": ["service=redis"] } S3. Establish a configuration information synchronization mechanism between custom Consul service components and external databases; 1. Dual-mechanism synchronization scheme Main scheme (Binlog listener): Maxwell is used to parse MySQL Binlog and push changes to the prometheus_configs table to Kafka in real time.

[0050] Custom service consumes messages and updates memory cache with a delay of <100ms.

[0051] 2. Alternative solutions (polling): # Environment variable controls the polling interval (default 60 seconds) SYNC_INTERVAL=30 custom-consul-service S4: Based on a custom Consul service component, simulated API interface, and configuration information synchronization mechanism, Prometheus enables dynamic management of the configuration information of the monitored target.

[0052] Prometheus configuration example scrape_configs: - job_name: 'dynamic_targets' consul_sd_configs: - server: 'custom-consul:8500' # Points to a custom service services: ['node_exporter']# Filter specific tasks tags: ['env=prod']# Filter by tag scrape_interval: 15s # Override global configuration Technical effect verification Testing in a Docker cluster (scale: 10 nodes):

[0053] Example 3: See Figure 2 As shown, this embodiment is a containerized Prometheus configuration persistence and real-time synchronization system, including... The configuration storage module is used to extract the Prometheus monitoring target configuration information from the Prometheus configuration file, persist the monitoring target configuration information to an external database, and output the persistent monitoring target configuration information. A custom service module is used to develop and deploy custom Consul service components in a containerized manner, and to call the persistent monitoring target configuration information output by the configuration stripping and persistence module to build a simulated API interface; The configuration information synchronization module is used to establish a configuration information synchronization mechanism between the custom Consul service component module and the external database; The dynamic discovery configuration module is used to set the service discovery method of consul_sd_configs, point the service address of the service discovery method to the custom Consul service component module, and trigger Prometheus to call the simulated API interface. Combined with the configuration information synchronization mechanism, it enables Prometheus to dynamically manage the configuration information of the monitoring target.

[0054] The technical features of this invention not described can be implemented by or using existing technology, and will not be repeated here. Of course, the above description is not a limitation of this invention, and this invention is not limited to the examples above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of this invention should also be within the protection scope of this invention.

Claims

1. A containerized method for Prometheus configuration persistence and real-time synchronization, characterized in that, Includes the following steps: Extract the Prometheus monitoring target configuration information from the Prometheus configuration file and persist it to an external database to obtain persistent monitoring target configuration information; the persistent monitoring target configuration information includes at least the monitoring task name and the address list of the monitored targets; Develop and deploy a custom Consul service component, and use the custom Consul service component to call persistent monitoring target configuration information to build a simulated API interface; the custom Consul service component is used to simulate the HTTP API interface of the Consul service. Establish a configuration information synchronization mechanism between custom Consul service components and external databases; Prometheus achieves dynamic management of monitoring target configuration information by using a custom Consul service component, a simulated API interface, and a configuration information synchronization mechanism.

2. The containerized Prometheus configuration persistence and real-time synchronization method according to claim 1, characterized in that, The persistent monitoring target configuration information also includes the capture interval, timeout period, indicator path, protocol type, and tags.

3. The containerized Prometheus configuration persistence and real-time synchronization method according to claim 2, characterized in that, The custom Consul service component is deployed in a containerized manner.

4. The containerized Prometheus configuration persistence and real-time synchronization method according to claim 3, characterized in that, The simulation API interface includes GET / v1 / agent / services: Returns a list of all monitoring tasks and monitored targets' addresses that are persistently stored in an external database; GET / v1 / health / service / {service_name}: Returns the health status data of the monitored target based on the monitoring task name; GET / v1 / agent / self: Returns the preset Agent metadata; GET / v1 / catalog / services: Returns the mapping relationship between all monitoring tasks and their tags; PUT / v1 / agent / service / register: Receives external registration requests and writes the data to an external database.

5. The containerized Prometheus configuration persistence and real-time synchronization method according to claim 4, characterized in that, The established configuration information synchronization mechanism adopts any one of the following methods: listening to change notifications from external databases, periodically polling persistently monitoring target configuration information, parsing binary logs or change data streams from external databases.

6. The containerized Prometheus configuration persistence and real-time synchronization method according to claim 5, characterized in that, The method of dynamically managing the configuration information of the monitoring target using Prometheus based on a custom Consul service component, a simulated API interface, and a configuration information synchronization mechanism includes setting the service discovery method of consul_sd_configs in the Prometheus configuration file and pointing the service address to the custom Consul service component. Prometheus calls the simulated API interface and combines it with the configuration information synchronization mechanism to achieve dynamic management of the configuration information of the monitoring target.

7. The containerized Prometheus configuration persistence and real-time synchronization method according to claim 6, characterized in that, The configuration of the consul_sd_configs service discovery method includes configuring the network address and port of the custom service, optional monitoring task filtering parameters, and optional tag filtering parameters.

8. The containerized Prometheus configuration persistence and real-time synchronization method and system according to claim 7, characterized in that, The external databases include relational databases and key-value databases.

9. A containerized Prometheus configuration persistence and real-time synchronization system, characterized in that, include The configuration storage module is used to extract the Prometheus monitoring target configuration information from the Prometheus configuration file, persist the monitoring target configuration information to an external database, and output the persistent monitoring target configuration information. A custom service module is used to develop and deploy custom Consul service components in a containerized manner, and to call the persistent monitoring target configuration information output by the configuration stripping and persistence module to build a simulated API interface; The configuration information synchronization module is used to establish a configuration information synchronization mechanism between the custom Consul service component module and the external database; The dynamic discovery configuration module is used to set the service discovery method of consul_sd_configs, point the service address of the service discovery method to the custom Consul service component module, and trigger Prometheus to call the simulated API interface. Combined with the configuration information synchronization mechanism, it enables Prometheus to dynamically manage the configuration information of the monitoring target.

Images