A method, system, device, and storage medium for synchronizing Prometheus alarm data.

Abstract

This invention discloses a method, system, device, and storage medium for synchronizing Prometheus alarm data, belonging to the field of database synchronization technology. It addresses the technical problem of efficiently managing and storing historical alarm data. The method involves processing synchronization service data pushed from Prometheus webhooks and actively pulled from HTTP interfaces. After processing, the system selects whether to add or update alarm data to MySQL based on the processing results. During the addition or update process, memory caching and Redis are used as secondary caches to reduce the pressure on MySQL. This invention, based on existing MySQL and Redis components, solves the problems of Prometheus alarm data aggregation, sorting, and pagination. This solution does not require the introduction of new components, can be deployed in a distributed manner, and in a distributed environment, there is no need to consider data consistency issues. The architecture is simple and easy to maintain.

Description

Technical Field

[0001] This invention relates to the field of database synchronization technology, and more specifically, to a method, system, device, and storage medium for synchronizing Prometheus alarm data. Background Technology

[0002] In modern, complex network systems, monitoring and alerting are essential components. Prometheus, a popular open-source monitoring tool, offers rich alerting features, allowing users to manage alert rules through AlertManager and promptly notify maintenance personnel.

[0003] However, in large-scale distributed systems, Prometheus monitors a massive number of metrics, generating a correspondingly large number of alerts daily. While it provides powerful data storage and query capabilities (PromQL), it struggles to meet flexible and ever-changing business needs, such as aggregating, sorting, and paginating historical data from multiple monitoring metrics. Therefore, efficiently managing and storing historical alert data becomes a crucial issue to address these flexible business requirements. Currently, some solutions exist, such as storing historical alert data in Elasticsearch or InfluxDB. However, these solutions require deploying separate databases or using third-party components, increasing system complexity and maintenance costs. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art. The purpose of the present invention is to provide a method for synchronizing Prometheus alarm data.

[0005] The second objective of this invention is to provide a system for synchronizing Prometheus alarm data.

[0006] The third objective of this invention is to provide a computer device.

[0007] The fourth objective of this invention is to provide a computer storage medium.

[0008] To achieve the first objective mentioned above, this invention provides a method for synchronizing Prometheus alarm data. The method processes the synchronization service data pushed from Prometheus webhook and actively pulled from the HTTP interface. After data processing, the method selects whether to add or update alarm data to MySQL based on the processing results. During the addition or update process, memory caching and Redis are used as secondary caching to reduce the pressure on MySQL during the addition or update process.

[0009] As a further improvement, the following steps are included:

[0010] Step S1. Configure the webhook address. Add the webhook address to all Prometheus monitoring items. When an alarm is triggered, Prometheus will push the alarm data to the synchronization service interface.

[0011] Step S2. Parse alarm data. When the synchronization service receives alarm data from the webhook or actively retrieves it, it parses out the relevant labels, startsAt, endsAt, and status key information from the data. Labels can identify a certain monitoring item, startsAt can identify a certain alarm for that monitoring item, and endsAt is a future time. If there are no new alarms after endsAt, it means that the monitoring item has returned to normal.

[0012] Step S3. Alarm Data Storage: To facilitate querying and storage, the labels data parsed in Step S2 is sorted, and its hash value, labels_hash, is taken. Labels_hash and startsAt are used as query conditions to query the database. If no alarm is found, it indicates a newly triggered alarm, and labels_hash, startsAt, endsAt, and status are inserted as a new record into the MySQL database. If an alarm is found, it indicates a continuing alarm. If endsAt is greater than the old value, the endsAt and status information in the record are updated; otherwise, no action is taken.

[0013] Step S4. Caching strategy: In order to reduce the pressure on the database, a two-level cache is implemented in the process of step S3, that is, local memory and Redis are used as L1 and L2 caches respectively.

[0014] Step S5. Distribute the alarm data in a distributed manner.

[0015] Furthermore, step S4 includes step S41: the cached query process.

[0016] If no cache hit occurs, write labels_hash+startsAt as the key and endsAt as the value to the cache, in the order of L1 and L2.

[0017] If the L1 cache is hit, and endsAt>value, update the endsAt field in the corresponding record in the database. After success, update the L1 and L2 caches in sequence. Otherwise, the process ends directly.

[0018] L2 cache hit. When endsAt > value, update the endsAt field in the corresponding record of the database. After success, update the L1 and L2 caches in sequence. In other cases, directly end the process.

[0019] Further, step S42 is included in step S4: cache expiration handling.

[0020] L1 cache expiration handling. The synchronization service periodically traverses the in-memory cache keys. When value is less than the current time, it means the cache has expired and can be cleared.

[0021] L2 cache expiration handling. By setting the expireAt parameter with the value of endsAt, let Redis automatically manage the cache expiration and cleaning.

[0022] Further, step S43 is included in step S4: handling cache consistency in case of exceptions.

[0023] ①. After the database operation is successful, before writing to the L1 cache, if the system crashes or restarts and the L1 cache has been cleared, at this time the L2 cache is still the old value. Since it is necessary to judge cache value < endsAt when the cache hits, and endsAt is a time data that only increases and does not decrease, this situation has no impact on data consistency and the cache becomes invalid.

[0024] ②. After the database operation is successful, when the system crashes before writing to the L2 or the write to the L2 cache fails, at this time the L1 is still valid, the L2 layer is the old value, and the cache is still valid. The situation after the system crashes and restarts is the same as in step ①.

[0025] Further, step S51 is included in step S5: optimization of inconsistent L1 caches in multiple nodes. In the case of distributed deployment, there must be distributed data consistency problems between nodes. However, due to the characteristics of the continuously increasing endsAt of the alarm data, the cache layer data consistency does not affect the synchronization effect, but only affects the cache effect. Therefore, when appending the webhook address to the monitoring item, bring the labels_hash in the query parameter, and perform directional routing according to the labels_hash parameter at the LB layer to ensure that the data push requests for the same alarm item always fall on the same node, increasing the role of the L1 cache.

[0026] Furthermore, step S5 includes step S52, which optimizes the processing of multiple nodes repeatedly. In step S51, the problem of pushing new alarms has been solved by directing the route according to labels_hash. However, when actively pulling alarm data, there may be multiple cases of duplicate alarm data. Therefore, the same strategy as LB using labels_hash can be maintained to filter alarm data processing tasks that do not belong to this node and avoid duplicate calculations.

[0027] To achieve the second objective mentioned above, the present invention provides a system for synchronizing Prometheus alarm data, comprising:

[0028] The configuration module is used to configure the webhook address, appending the webhook address to all Prometheus monitoring items. When an alarm is triggered, Prometheus will push alarm data to the synchronization service interface.

[0029] The parsing module is used to parse alarm data. When the synchronization service receives alarm data from webhooks or actively fetches it, it parses relevant labels, startsAt, endsAt, and status information from the data. Labels can identify a specific monitoring item, startsAt can identify a specific alarm for that monitoring item, and endsAt is a future time. If there are no new alarms after endsAt, it means that the monitoring item has returned to normal.

[0030] The storage module is used for alarm data storage. To facilitate querying and storage, the labels data parsed in step S2 is sorted and its hash value, labels_hash, is taken. Labels_hash and startsAt are used as query conditions to query the database. If no result is found, it means that the alarm has been newly triggered, and labels_hash, startsAt, endsAt, and status are inserted into the MySQL database as a new record. If a result is found, it means that the alarm is ongoing. If endsAt is greater than the old value, the endsAt and status information in the record are updated; otherwise, no action is taken.

[0031] The caching module is used to perform secondary caching in step S3, that is, local memory and Redis are used as L1 and L2 caches respectively, in order to reduce the pressure on the database;

[0032] The deployment module is used for distributed deployment of alarm data.

[0033] To achieve the third objective mentioned above, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned method for synchronizing Prometheus alarm data.

[0034] To achieve the fourth objective mentioned above, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method for synchronizing Prometheus alarm data.

[0035] Beneficial effects

[0036] Compared with the prior art, the advantages of this invention are as follows:

[0037] This invention provides a solution for synchronizing Prometheus alarm history using existing MySQL and Redis components, resolving issues related to Prometheus alarm data aggregation, sorting, and pagination. This solution does not require the introduction of new components, can be deployed in a distributed manner, and eliminates the need to consider data consistency issues in a distributed environment. Its architecture is simple and easy to maintain. Attached Figure Description

[0038] Figure 1 This is a flowchart of the process of the present invention. Detailed Implementation

[0039] The present invention will be further described below with reference to specific embodiments shown in the accompanying drawings.

[0040] See Figure 1 This method synchronizes Prometheus alert data by processing data pushed from Prometheus webhooks and actively pulled from HTTP interfaces. After processing, the method selects whether to add or update alert data to MySQL based on the processing results. During the addition or update process, memory caching and Redis are used as secondary caches to reduce the pressure on MySQL.

[0041] Specifically, the following steps are included:

[0042] Step S1. Configure the webhook address. Add the webhook address to all Prometheus monitoring items. That is, the interface address of the synchronization service. When an alarm is triggered, Prometheus will push alarm data to the interface of the synchronization service.

[0043] Step S2. Parse alarm data. When the synchronization service receives alarm data from the webhook or actively retrieves it, it parses out the relevant labels, startsAt, endsAt, and status key information from the data. Labels can identify a certain monitoring item, startsAt can identify a certain alarm for that monitoring item, and endsAt is a future time. If there are no new alarms after endsAt, it means that the monitoring item has returned to normal.

[0044] Step S3. Alarm Data Storage: To facilitate querying and storage, the labels data parsed in Step S2 is sorted, and its hash value, labels_hash, is taken. Labels_hash and startsAt are used as query conditions to query the database. If no result is found, it indicates a newly triggered warning, and labels_hash, startsAt, endsAt, and status are inserted as a new record into the MySQL database. Other data fields can be added according to business needs. If a result is found, it indicates a continuing warning. If endsAt is greater than the old value, the endsAt and status information in the record are updated; otherwise, no action is taken.

[0045] Step S4. Caching strategy: In order to reduce the pressure on the database, a two-level cache is implemented in the process of step S3, that is, local memory and Redis are used as L1 and L2 caches respectively.

[0046] Step S5. Distribute the alarm data in a distributed manner.

[0047] Step S4 includes the following steps:

[0048] Step S41: The cached query process,

[0049] If no cache hit occurs, write labels_hash+startsAt as the key and endsAt as the value to the cache, in the order of L1 and L2.

[0050] If the L1 cache is hit, and endsAt>value, update the endsAt field in the corresponding record in the database. After success, update the L1 and L2 caches in sequence. Otherwise, the process ends directly.

[0051] When the L2 cache is hit, if endsAt > value, the endsAt field in the corresponding record in the database is updated. After success, the L1 and L2 caches are updated in sequence. Otherwise, the process ends directly.

[0052] Step S42: Cache expiration handling,

[0053] L1 cache expiration handling. The synchronization service periodically traverses the in-memory cache keys. When the value is less than the current time, it indicates that the cache has expired and can be cleared.

[0054] L2 cache expiration handling. By setting the expireAt parameter with the value endsAt, Redis automatically manages the expiration and cleaning of the cache.

[0055] Step S43: Handling cache consistency in exceptional cases

[0056] ①. After a successful database operation, before writing to the L1 cache, if the system crashes or restarts and the L1 cache has been cleared, the L2 cache still contains the old value at this time. Since it is necessary to check if the cache value < endsAt when the cache is hit, and endsAt is a time data that only increases and does not decrease, this situation has no impact on data consistency and the cache becomes invalid. That is, before the crash recovery, new warnings are generated.

[0057] ②. After a successful database operation, if the system crashes before writing to the L2 or the write to the L2 cache fails, the L1 cache is still valid at this time, and the L2 layer contains the old value, and the cache is still valid. The situation after the system crashes or restarts is the same as in step ①.

[0058] In step S5, it includes step S51, optimization of multi-node L1 cache inconsistency. In a distributed deployment scenario, there must be distributed data consistency issues between nodes, that is, the consistency issue of the L1 layer cache. However, due to the continuously increasing characteristic of the alarm data endsAt, the cache layer data consistency does not affect the synchronization effect, but only affects the cache effect. Therefore, when appending the webhook address to the monitoring item, bring the labels_hash in the query parameter, and perform directional routing at the LB layer according to the labels_hash parameter to ensure that the data push requests for the same alarm item always fall on the same node, enhancing the role of the L1 cache.

[0059] In step S5, it includes step S52, optimization of multi-node duplicate processing. In step S51, the problem of new alarm push has been solved by directional routing according to labels_hash. However, in the case of actively pulling alarm data, there may be a situation where multiple alarm data are repeated. Therefore, the same strategy as the LB using labels_hash can be maintained to filter the alarm data processing tasks that do not belong to this node and avoid duplicate calculations.

[0060] A system for synchronizing Prometheus alarm data, including:

[0061] The configuration module is used to configure the webhook address, appending the webhook address to all Prometheus monitoring items. When an alarm is triggered, Prometheus will push alarm data to the synchronization service interface.

[0062] The parsing module is used to parse alarm data. When the synchronization service receives alarm data from webhooks or actively fetches it, it parses relevant labels, startsAt, endsAt, and status information from the data. Labels can identify a specific monitoring item, startsAt can identify a specific alarm for that monitoring item, and endsAt is a future time. If there are no new alarms after endsAt, it means that the monitoring item has returned to normal.

[0063] The storage module is used for alarm data storage. To facilitate querying and storage, the labels data parsed in step S2 is sorted and its hash value, labels_hash, is taken. Labels_hash and startsAt are used as query conditions to query the database. If no result is found, it means that the alarm has been newly triggered, and labels_hash, startsAt, endsAt, and status are inserted into the MySQL database as a new record. If a result is found, it means that the alarm is ongoing. If endsAt is greater than the old value, the endsAt and status information in the record are updated; otherwise, no action is taken.

[0064] The caching module is used to perform secondary caching in step S3, that is, local memory and Redis are used as L1 and L2 caches respectively, in order to reduce the pressure on the database;

[0065] The deployment module is used for distributed deployment of alarm data.

[0066] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method for synchronizing Prometheus alarm data.

[0067] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for synchronizing Prometheus alarm data.

[0068] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention, and these will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. A method for synchronizing Prometheus alarm data, characterized in that, The system processes synchronous service data pushed from Prometheus webhooks and actively pulled from the HTTP interface. After data processing, it decides whether to add or update alarm data to MySQL based on the processing results. During the addition or update process, it uses memory caching and Redis as a second-level cache to reduce the pressure on MySQL during the addition or update process. Specifically, the following steps are included: Step S1. Configure the webhook address. Add the webhook address to all Prometheus monitoring items. When an alarm is triggered, Prometheus will push the alarm data to the synchronization service interface. Step S2. Parse alarm data. When the synchronization service receives alarm data from the webhook or actively retrieves it, it parses out the relevant labels, startsAt, endsAt, and status key information from the data. Labels can identify a certain monitoring item, startsAt can identify a certain alarm for that monitoring item, and endsAt is a future time. If there are no new alarms after endsAt, it means that the monitoring item has returned to normal. Step S3. Alarm Data Storage: To facilitate querying and storage, the labels data parsed in Step S2 is sorted, and its hash value, labels_hash, is taken. Labels_hash and startsAt are used as query conditions to query the database. If no alarm is found, it indicates a newly triggered alarm, and labels_hash, startsAt, endsAt, and status are inserted as a new record into the MySQL database. If an alarm is found, it indicates a continuing alarm. If endsAt is greater than the old value, the endsAt and status information in the record are updated; otherwise, no action is taken. Step S4. Caching strategy: In order to reduce the pressure on the database, a two-level cache is implemented in the process of step S3, that is, local memory and Redis are used as L1 and L2 caches respectively. Step S5. Distribute the alarm data in a distributed manner.

2. The method for synchronizing Prometheus alarm data according to claim 1, characterized in that, Step S4 includes step S41: the cached query process. If no cache hit occurs, write labels_hash+startsAt as the key and endsAt as the value to the cache, in the order of L1 and L2. If the L1 cache is hit, and endsAt>value, update the endsAt field in the corresponding record in the database. After success, update the L1 and L2 caches in sequence. Otherwise, the process ends directly. When the L2 cache is hit, if endsAt > value, the endsAt field in the corresponding record in the database is updated. After success, the L1 and L2 caches are updated in sequence. Otherwise, the process ends directly.

3. The method for synchronizing Prometheus alarm data according to claim 2, characterized in that, Step S4 includes step S42: cache expiration handling. L1 cache expiration handling: The synchronization service periodically traverses the memory cache key. When the value is less than the current time, it means that the cache has expired and can be cleared. L2 cache expiration handling. By setting the expireAt parameter with the value of endsAt, Redis automatically manages the expiration and cleaning of the cache.

4. The method for synchronizing Prometheus alarm data according to claim 3, characterized in that, In step S4, it includes step S43: handling cache consistency in case of exceptions. ①. After a successful database operation, before writing to the L1 cache, if the system crashes or restarts and the L1 cache has been cleared, the L2 cache still has the old value at this time. Since it is necessary to judge whether the cache value < endsAt when the cache is hit, and endsAt is a time data that only increases and does not decrease, this situation has no impact on data consistency and the cache becomes invalid. ②. After a successful database operation, if the system crashes before writing to the L2 or the write to the L2 cache fails, the L1 cache is still valid at this time, and the L2 layer has the old value, and the cache is still valid. The situation after the system crashes and recovers or restarts is the same as in step ①.

5. The method for synchronizing Prometheus alarm data according to claim 1, characterized in that, In step S5, it includes step S51, optimization of multi-node L1 cache inconsistency. In a distributed deployment scenario, there must be distributed data consistency problems between nodes. However, due to the characteristic that the alarm data endsAt keeps increasing, the data consistency of the cache layer does not affect the synchronization effect, but only affects the cache effect. Therefore, when adding a webhook address to a monitoring item, the labels_hash is carried in the query parameter, and the LB layer makes a directed route according to the labels_hash parameter to ensure that the data push requests for the same alarm item always fall on the same node, increasing the role of the L1 cache.

6. The method for synchronizing Prometheus alarm data according to claim 5, characterized in that, In step S5, it includes step S52, optimization of multi-node duplicate processing. In step S51, the problem of new alarm pushing has been solved by directed routing according to labels_hash. However, in the case of actively pulling alarm data, there may be a situation where multiple alarm data are repeated. Therefore, the same strategy as the LB using labels_hash can be maintained to filter out the alarm data processing tasks that do not belong to this node and avoid duplicate calculations.

7. A system for synchronizing Prometheus alarm data, characterized in that, It includes: A configuration module for configuring the webhook address, adding the webhook address to all Prometheus monitoring items. When an alarm is triggered, Prometheus will push the alarm data to the interface of the synchronization service. An analysis module for analyzing the alarm data. When the synchronization service receives the alarm data from the webhook or actively pulled, it analyzes the relevant labels, startsAt, endsAt, and status key information from the data. Labels can determine a certain monitoring item, startsAt can determine a certain alarm of the monitoring item, endsAt is a future time. When there is no new alarm after endsAt, it means that the monitoring item has returned to normal. The storage module is used for alarm data storage. To facilitate querying and storage, the labels data parsed in step S2 is sorted and its hash value, labels_hash, is taken. Labels_hash and startsAt are used as query conditions to query the database. If no result is found, it means that the alarm has been triggered. In this case, labels_hash, startsAt, endsAt, and status are inserted as a new record into the MySQL database. If a result is found, it means that the alarm is ongoing. If endsAt is greater than the old value, the endsAt and status information in the record are updated. Otherwise, no action will be taken; The caching module is used to perform secondary caching in step S3, that is, local memory and Redis are used as L1 and L2 caches respectively, in order to reduce the pressure on the database; The deployment module is used for distributed deployment of alarm data; The specific working process of the system is as follows: Configure the webhook address to add the webhook address to all Prometheus monitoring items. When an alarm is triggered, Prometheus will push the alarm data to the synchronization service interface. When the alarm data is received from the webhook or actively pulled alarm data, the synchronization service extracts relevant labels, startsAt, endsAt and status information from the data. Labels can identify a specific monitoring item, startsAt can identify a specific alarm for that monitoring item, and endsAt is a future time. If there are no new alarms after endsAt, it means that the monitoring item has returned to normal. For alarm data storage, to facilitate querying and storage, the labels data parsed in step S2 is sorted and its hash value, labels_hash, is taken. Labels_hash and startsAt are used as query conditions to query the database. If no alarm is found, it indicates a newly triggered warning, and labels_hash, startsAt, endsAt, and status are inserted as a new record into the MySQL database. If an alarm is found, it indicates an ongoing warning. If endsAt is greater than the old value, the endsAt and status information in the record are updated; otherwise, no action is taken. The caching strategy is to reduce the pressure on the database by implementing a two-level cache in step S3, namely, local memory and Redis are used as L1 and L2 caches, respectively. Distribute alarm data in a distributed manner.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements a method for synchronizing Prometheus alarm data as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a method for synchronizing Prometheus alarm data as described in any one of claims 1-6.

Images