Method and system for solving incorrect current value of alarm recovery data of monitoring system

Abstract

The present application relates to a method and system for solving the problem of incorrect current value of alarm recovery data in a monitoring system, the method comprising the following steps: S1: receiving alarm information sent by an alarm unit through a configured http interface; S2: determining whether the alarm information is alarm recovery information, if yes, proceeding to S3, otherwise, directly proceeding to S4; S3: calling an api interface of a prometheus server to query current value information of a monitored index, and then writing the query result into alarm metadata; S4: sending the alarm information to a specified terminal through a webhook. The method combines the customizable feature of the annotations attribute in the alarm rule with the webhook function of the alertmanager and the query api of the prometheus server to realize the function of automatically querying the current value of the monitored index during alarm recovery, and solves the problem of incorrect current value of the recovery information of the prometheus itself.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to computer monitoring technology. Background Technology

[0002] Prometheus is an open-source monitoring solution that provides functions such as collecting, storing, querying, and alerting monitoring metrics. As a graduation project of the Cloud Native Computing Foundation (CNCF), Prometheus has been widely adopted in the cloud-native field and has gradually become one of the most popular monitoring solutions in the industry.

[0003] The following is an example of Prometheus alert rule configuration: - alert: MyAlert expr: foo > 100 for: 1m annotations: description: 'Foo is greater than 100'. When the foo metric is detected to be greater than 100, Prometheus will issue an alert. However, when foo is less than or equal to 100, the recovery message issued by Prometheus does not include the current value of the foo metric, but rather the data from the last alert message. This problem is due to the alert mechanism itself and cannot be resolved internally. Therefore, during alert recovery, the alert receiver cannot know the current value of the alert metric and thus cannot determine whether further manual intervention is required. Summary of the Invention

[0004] One objective of this invention is to provide a method for resolving the issue of incorrect current values ​​in alarm recovery data of a monitoring system, thereby addressing the problem that the Prometheus monitoring system cannot obtain the current values ​​of the monitored indicators in its alarm recovery information; another objective is to provide a system for resolving the issue of incorrect current values ​​in alarm recovery data of a monitoring system.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: The method for resolving incorrect current values ​​of alarm recovery data in a monitoring system is as follows: S1: Receive alarm information sent by the alarm unit through the configured HTTP interface; S2: Determine if it is an alarm recovery message. If yes, proceed to S3; otherwise, proceed directly to S4. S3: Call the Prometheus server's API interface to query the current value information of the monitored metric, and then rewrite the query result into the alarm metadata; S4: Send alarm information to the specified terminal via webhook.

[0006] Based on the above technical means, since Prometheus itself cannot automatically obtain the current value information of the monitored metric when sending alarm recovery information, the above technical means receive the alarm information sent by the alarm unit through the configured HTTP interface, and query the current value information of the monitored metric by calling the API interface. The query result is rewritten into the alarm data, and then sent to the specified terminal through the webhook set on the Prometheus server, thereby overcoming the problem of not being able to obtain the current value of the monitored metric.

[0007] Furthermore, the Prometheus server is configured with alarm rules, and the annotations of the alarm rules support the configuration of custom attributes. The annotations have custom fields that represent the query expression when the alarm is recovered. If the status field of the current alarm metadata is a field that represents the query expression when the alarm is recovered, then it is alarm recovery information.

[0008] Based on the above technical means, the customizable annotations attribute in alarm rules is utilized in conjunction with the webhook function of AlertManager and the query API of Prometheus Server to realize the function of automatically querying the current value of monitored metrics when alarms are restored, thus solving the problem of incorrect current values ​​of recovery information in Prometheus itself.

[0009] Furthermore, the query expression for alarm recovery is configured with N label variables, where N is a natural number greater than 0. When the alarm unit issues alarm information, and the alarm information is alarm recovery information, the label variables will be automatically replaced with the actual values.

[0010] The above technical methods ensure that the query results are unique.

[0011] Furthermore, the naming rule for the Label variable is {{ $labels.label_name}}, where label_name is the label name of the specific monitored metric.

[0012] Furthermore, the method of rewriting the query results into the alarm metadata specifically involves: retrieving the field value of the query expression representing the alarm recovery from the annotations, calling the API interface to query the current value information of the monitored metric, represented by real_current_value, and replacing the current_value in the alarm metadata with real_current_value.

[0013] Furthermore, the designated terminal can be one or more of the following: SMS, email, WeChat Work, or DingTalk.

[0014] A system for resolving incorrect current values ​​of alarm recovery data in a monitoring system based on the above method includes: The alarm information receiving module is configured to receive alarm information sent by the alarm unit through the configured HTTP interface; The judgment module is configured to determine whether the alarm information is alarm recovery information; The query module is configured to call the Prometheus server's API interface to query the current value information of the monitored metrics, and then rewrite the query results into the alarm metadata. The sending module is configured to send alarm information to a specified terminal via a webhook.

[0015] Furthermore, if the judgment module determines that the current alarm metadata's status field value is a field representing the query expression during alarm recovery, then it is alarm recovery information. The Prometheus server is configured with alarm rules, and the annotations of the alarm rules support the configuration of custom attributes. The annotations have custom fields representing the query expression during alarm recovery.

[0016] Furthermore, the query expression for alarm recovery is configured with N label variables, where N is a natural number greater than 0. When the alarm unit issues alarm information, and the alarm information is alarm recovery information, the label variables will be automatically replaced with the actual values.

[0017] Furthermore, the field value representing the query expression during alarm recovery is extracted from the annotations, and the API interface is called to query the current value information of the monitored metric, represented by real_current_value. The current_value in the alarm metadata is replaced with real_current_value.

[0018] The beneficial effects of this invention are: This invention utilizes the customizable annotations attribute in alarm rules and combines it with the webhook function of AlertManager and the query API of Prometheus Server to automatically query the current value of monitored metrics during alarm recovery, thus solving the problem of incorrect current values ​​in Prometheus's own recovery information. Attached Figure Description

[0019] Figure 1 The logical architecture diagram of Example 1 Figure 2 This is a logic diagram of the current value in the corrected alarm recovery information of Example 1; Figure 3 This is a structural diagram of the system in Example 2.

[0020] Among them, 1-alarm information receiving module; 2-judgment module; 3-query module; 4-sending module. Detailed Implementation

[0021] The following description, with reference to the accompanying drawings and preferred embodiments, illustrates the implementation of the technical solution of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0022] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0023] Example 1 This embodiment proposes a method to resolve the issue of incorrect current values ​​in alarm recovery data of a monitoring system. In the Prometheus monitoring system, alarm messages are uniformly processed by AlertManager. A webhook can be configured in AlertManager to send alarm information to a specified API, enabling customized processing of alarm information. Although Prometheus itself cannot automatically obtain the current value of the monitored metric when sending alarm recovery information, the webhook function configured in AlertManager can be used to actively query the current value of the monitored metric by calling the API provided by the Prometheus server when alarm recovery information is received. The query result is then rewritten into the alarm data and sent to the alarm recipient.

[0024] Specifically, such as Figure 1 and Figure 2 As shown, the method is as follows: S1: Receive alarm information sent by the alarm unit through the configured HTTP interface.

[0025] Alertmanager supports sending alert information to a specified HTTP interface via webhook, so alert information can be sent to this program to implement subsequent custom processing logic.

[0026] S2: Determine if it is an alarm recovery message. If yes, proceed to S3; otherwise, proceed directly to S4.

[0027] In this step, it is determined whether the value of the `status` field in the current alarm metadata is a field representing the query expression used for alarm recovery. If it is, then it is alarm recovery information; otherwise, it is not. In this embodiment, the Prometheus server is configured with alarm rules, and the annotations of the alarm rules support configuring custom attributes. These annotations include a field representing the query expression used for alarm recovery. S3: Call the Prometheus server's API to query the current value information of the monitored metric, and then rewrite the query results into the alarm metadata.

[0028] After calculating the query expression for the monitored metric, the query interface provided by the Prometheus server needs to be called to retrieve the current value data of the metric and rewrite the query result into the alarm metadata.

[0029] S4: Send alarm information to the specified terminal via webhook.

[0030] Since AlertManager sends all alert information to the webhook, this program needs to implement subsequent alert sending functions, such as SMS, email, WeChat Work, DingTalk, etc.

[0031] In this embodiment, the alarm rules are configured as follows: Alarm rules in the Prometheus monitoring system are configured in YAML format within the Prometheus server. The core attribute is the `expr` field, which is essentially a Prometheus custom query language (PromQL). Custom attributes can also be configured in the annotations of the alarm rules. This feature allows the alarm recovery query expression `resolved_expr` to be configured in the annotations along with the alarm rules. This query expression (`resolved_expr`) requires 0 to N label variables to ensure that only one record is retrieved when the Prometheus server query interface is called.

[0032] The naming convention for Label variables is {{ $labels.label_name}}, where label_name is the label name of the specific monitoring metric. The Prometheus server will automatically replace the variable with the actual value when sending alarm information.

[0033] The number of Label variables needs to be configured in conjunction with the expr expression. By using 0 to N variables, the range of query results of the query expression can be narrowed down to exactly one data point.

[0034] In this embodiment, the method described above is explained using the monitoring of hard disk usage on a Linux server as an example. The implementation steps are as follows: Configure alarm rules as follows: Alert: Hard drive usage is greater than 90%. expr: 100 - (node_filesystem_free_bytes / node_filesystem_size_bytes* 100) > 90 for: 1m annotations: Description: "Disk usage of $labels.device on server {{ $labels.instance}} is greater than 90%". resolved_expr: "100 - (node_filesystem_free_bytes{device='{{ $labels.device}}',instance='{{ $labels.instance}}'} / node_filesystem_size_bytes{device='{{ $labels.device}}',instance='{{ $labels.instance}}'} *100)" current_value: "{{ $value}}" In this alert rule, an alert message will be sent to AlertManager when the utilization of any disk on the disk server exceeds 90%. Three custom fields have been added to the annotations, among which the resolved_expr field represents the query expression for alert recovery, and two label variables are configured in this expression: $labels.device: Represents the disk being monitored. $labels.instance: Represents the IP address of the monitored server. When the Prometheus server issues an alert, it automatically replaces these two variables with their actual values ​​by adding the corresponding label to the resolved_expr expression. The purpose is to ensure that the query results are unique when calling the Prometheus server query interface.

[0035] S1: Develop and implement an HTTP interface (e.g., http: / / 192.168.1.1 / api / custom-alerts-handle) to receive alarm information from AlertManager. Implement the logic for calculating the current value of the alarm recovery information within this interface. The calculation steps are as follows: In the example, when the server disk monitoring system recovers from an alarm, the alarm metadata sent is as follows: { "alerts": [ { "fingerprint": "002cb6fac24e2847", "annotations": { "description": "Disk usage of / dev / sdi1 on server 10.64.49.15:9100 is greater than 90%", "resolved_expr": "100 - (node_filesystem_free_bytes{device=' / dev / sdi1',instance='10.64.49.15:9100'} / node_filesystem_size_bytes{device=' / dev / sdi1',instance='10.64.49.15:9100'} * 100)", "current_value": "94.00011007119181" }, "startsAt": "2021-03-31T16:27:01.156150324Z", "endsAt": "2021-04-01T04:22:01.156150324Z", "status": "resolved", "labels": { "severity": "warn", "cluster": "prod", "instance": "10.64.49.15:9100", "kind": "bigdata", "alertname": "Hard disk usage is greater than 90%", "job": "node_exporter", "device": " / dev / sdi1", "fstype": "xfs", "mountpoint": " / u09 } } ] } S2: Determine whether the value of the status field in the current alarm metadata is resolved. If it is, it means that the alarm has recovered and needs to be processed in the next step. If not, proceed to step S4.

[0036] S3: Retrieve the resolved_expr field value from annotations and call the query interface provided by the Prometheus server to obtain the current value information of the monitored metric, represented as real_current_value.

[0037] Replace the current_value in the annotations attribute of the alarm metadata with real_current_value. S4: Configure a webhook address in AlertManager to send alert information via SMS, email, etc.

[0038] The method for configuring the webhook address is as follows: - name: 'web.hook' webhook_configs: - url: 'http: / / 192.168.1.1 / api / custom-alerts-handle' Example 2 This embodiment, based on Embodiment 1, proposes a system to resolve the issue of incorrect current values ​​in alarm recovery data from a monitoring system. Figure 3 As shown, the system includes: Alarm information receiving module 1 is configured to receive alarm information sent by alarm units through a configured HTTP interface; Module 2 is configured to determine whether the alarm information is alarm recovery information; Query module 3 is configured to call the Prometheus server's API interface to query the current value information of the monitored metrics, and then rewrite the query results into the alarm metadata. Sending module 4 is configured to send alarm information to a specified terminal via webhook.

[0039] The remaining contents are as described in Example 1, and will not be repeated here.

[0040] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A method for resolving incorrect current values ​​of alarm recovery data in a monitoring system, characterized by: The method is as follows: S1: Receive alarm information sent by the alarm unit through the configured HTTP interface; S2: Determine if it is an alarm recovery message. If yes, proceed to S3; otherwise, proceed directly to S4. S3: Call the Prometheus server's API interface to query the current value information of the monitored metric, and then rewrite the query result into the alarm metadata; S4: Send alarm information to the specified terminal via webhook; The Prometheus server is configured with alarm rules, and the annotations of the alarm rules support the configuration of custom attributes. The annotations have custom fields that represent the query expression when the alarm is recovered. If the status field of the current alarm metadata is a field that represents the query expression when the alarm is recovered, then it is alarm recovery information.

2. The method according to claim 1, characterized in that: The query expression for alarm recovery is configured with N label variables, where N is a natural number greater than 0. When the alarm unit issues an alarm message, and the alarm message is an alarm recovery message, the label variables will be automatically replaced with the actual values.

3. The method according to claim 2, characterized in that: The naming rule for the label variable is {{ $labels.label_name}}, where label_name is the label name of the specific monitored metric.

4. The method according to claim 2, characterized in that: The method for rewriting the query results into the alarm metadata is as follows: extract the field value of the query expression representing the alarm recovery from the annotations, call the API interface to query the current value information of the monitored metric, represented by real_current_value, and replace the current_value in the alarm metadata with real_current_value.

5. The method according to claim 1, characterized in that: The designated terminal is one or more of the following: SMS, email, WeChat Work, or DingTalk.

6. A system for resolving incorrect current values ​​of alarm recovery data in a monitoring system based on the method described in any one of claims 1-5, characterized in that: include: The alarm information receiving module is configured to receive alarm information sent by the alarm unit through the configured HTTP interface; The judgment module is configured to determine whether the alarm information is alarm recovery information; The query module is configured to call the Prometheus server's API interface to query the current value information of the monitored metrics, and then rewrite the query results into the alarm metadata. The sending module is configured to send alarm information to a specified terminal via a webhook.

7. The system according to claim 6, characterized in that: If the judgment module determines that the current alarm metadata's status field value is a field representing the query expression when the alarm is recovered, then it is alarm recovery information. The Prometheus server is configured with alarm rules, and the annotations of the alarm rules support the configuration of custom attributes. The annotations have custom fields representing the query expression when the alarm is recovered.

8. The system according to claim 6, characterized in that: The query expression for alarm recovery is configured with N label variables, where N is a natural number greater than 0. When the alarm unit issues an alarm message, and the alarm message is an alarm recovery message, the label variables will be automatically replaced with the actual values.

9. The system according to claim 6, characterized in that: Extract the field value of the query expression representing alarm recovery from the annotations, and call the API interface to query the current value information of the monitored metric, represented by real_current_value. Replace the current_value in the alarm metadata with real_current_value.

Images