A method and device for alarm management

Abstract

The present invention relates to the field of monitoring and operation and maintenance, and specifically provides an alarm management method and device. First, the alarm management module guides users to maintain alarm rules through the provided form page, including alarm rules, alarm shielding settings, and alarm message template settings; the alarm processing module continuously and cyclically processes and sends alarms according to the alarm rules, adds restrictions to the alarm information according to the configured rules, and then sends it to the monitoring system to obtain data, and shields, restores, silently judges, and sends the returned data. Compared with the prior art, the present invention can integrate the alarm information of multiple monitoring systems and realize the centralized management and display of the alarm information of multiple monitoring components.

Description

Technical Field

[0001] The present invention relates to the field of monitoring and operation and maintenance, and specifically provides an alarm management method and device. Background Art

[0002] The alarm function is an indispensable part of the modern operation and maintenance system. By monitoring the system's operating status and performance indicators in real time, the alarm monitoring system can quickly capture any abnormalities or potential problems and immediately trigger alarm notifications. This timely alarm mechanism ensures the stability of the system and the continuity of business, detects potential problems in a timely manner, and improves the stability and maintainability of the system.

[0003] The alarm monitoring system helps optimize resource allocation. By collecting and analyzing monitoring data, it can more accurately display the resource usage of the system, helping operation and maintenance personnel to make more reasonable resource allocation and expansion planning. The traditional alarm management method is for the monitoring system to collect status information and evaluate the monitoring data to trigger alarms, supplemented by built-in alarm components, and the alarm components send alarm notifications. For example, Prometheus uses the HTTP protocol to capture the status of the monitored components with a collection cycle of 5 seconds and evaluate the configured alarm rules every 10 seconds. When the conditions of an alarm rule are met, Prometheus switches the alarm status to pending. If the condition is still true in the next calculation cycle (after 10 seconds) and the duration meets the for duration defined in the rule (for example, 10 seconds), the alarm status will change to active and the alarm will be sent to the Alertmanager alarm component. After receiving the alarm, the Alertmanager sends the alarm notification to the corresponding recipient according to the configured routing rules. Different monitoring systems use different alarm components.

[0004] This traditional alarm management method has some disadvantages in actual processing:

[0005] 1. Complicated configuration and lack of intuitive display: The configuration process of alarm rules is relatively complicated. Users need to be familiar with many fields and settings. For example, the configuration file (alertmanager.yml) of the Alertmanager alarm component needs to define detailed rules such as routing tree information, grouping, suppression, and silence, which increases the user's learning cost and configuration difficulty. In addition, the display form of alarm information is single and lacks an intuitive visual interface. It is difficult for managers to quickly identify and understand the severity, distribution, and trend changes of alarms. There is a lack of functions such as alarm history statistics display, and users cannot conduct in-depth analysis or troubleshooting.

[0006] 2. Information fragmentation: Different monitoring systems have different alarm processing methods, and alarm information is usually scattered in various independent systems, making it difficult to manage in a unified manner. In large or complex monitoring environments, multiple instances need to be deployed to handle the alarm needs of different tenants, increasing the complexity of management and maintenance. These alarm components usually do not directly provide the function of long-term storage and query of historical alarm information, and cannot intuitively summarize alarm information, resulting in scattered alarm information. Summary of the invention

[0007] The present invention aims at solving the above-mentioned deficiencies of the prior art and provides an alarm management method with strong practicability.

[0008] A further technical task of the present invention is to provide an alarm management device that is rationally designed, safe and applicable.

[0009] The technical solution adopted by the present invention to solve its technical problem is:

[0010] An alarm management method, first, the alarm control module guides the user to maintain the alarm rules through the provided form page, including alarm rules, alarm shielding settings and alarm message template settings;

[0011] The alarm processing module processes and sends alarms in a continuous cycle according to the alarm rules. It adds restrictions to the alarm information according to the configured rules and then sends it to the monitoring system to obtain data. It also blocks, restores, silently judges and sends the returned data.

[0012] Furthermore, when setting alarm rules in the alarm management module, the alarm rules set a JavaScript expression. The user sets the expression to number>0, and the value of number is the number of data returned that meets the query conditions. An alarm will be issued only when the quantity condition meets the expression.

[0013] Furthermore, when setting up alarm shielding, the user configures the alarm shielding label, specifies the alarm shielding label key as the shielding field key, the alarm shielding operator as the shielding judgment standard, the alarm shielding value as the shielding field value, and sets the shielding time to control the shielding time. It supports periodic time shielding and fixed time shielding. The value of the periodic shielding time type muteTimeType is periodicMutes, and the number of weeks, period start time and stop time are set at the same time.

[0014] Furthermore, when setting the alarm message template, the message template is customized according to the response data. When the actual alarm message is sent, the fields are replaced according to the alarm information and the customized message format is sent.

[0015] Furthermore, in the alarm processing module, when adding restrictions to the data acquisition logic, the alarm shielding is implemented by adding fields before sending the data acquisition logic to the data source for the Clickhouse data source type alarm;

[0016] First, check whether the user has set the alarm shielding field. According to the alarm shielding tag configured by the user, use the tag name as the field, the tag operator as the judgment standard, and the tag value as the shielding value.

[0017] Furthermore, when making alarm shielding judgments on the returned data, for Prometheus type alarms, after the data source returns the alarm information, the returned alarm data is filtered and controlled according to the shielding event label in the current alarm configuration information to achieve alarm shielding. The label is matched through jsonpath, and alarms containing shielded fields will not be sent.

[0018] Furthermore, when judging the alarm recovery of the returned data, for the Prometheus data source type alarm, for the alarm event that has been generated, if you query it again through promql within the scheduling cycle and find that there is no returned content, it means that the current monitoring data no longer meets the threshold conditions specified in promql, indicating that the alarm has been restored;

[0019] The method for judging whether the alarm has been restored is to generate a unique HashCode based on the metric field value and the rule id, and compare the generated HashCode with the existing historical data of the current alarm in the alarm history information table. If the newly generated HashCode matches, it means that the alarm has not been restored;

[0020] If the newly generated HashCode cannot be matched in the database, it means that the alarm is a newly generated alarm information. If there is a HashCode in the database and the current alarm does not generate the same HashCode value, it means that the alarm has been restored.

[0021] Furthermore, when making an alarm silence judgment on the returned data, the alarm processing module matches the alarm data returned by the monitoring system according to the user-defined jsonpath silence condition, and compares the current alarm silence times with the preset threshold of alarm silence according to the alarm silence settings and the records of the alarm sending log. The time of the last alarm sending and the current time are calculated through the preset silence time. If an alarm in the alarm silence period is judged to be an unrecovered alarm, the same alarm message will not be sent to the user again.

[0022] Furthermore, when an alarm is sent for the returned data, the alarm information that is not blocked and is not in a silent period will have its message content format converted according to the message template configured by the user, and the alarm will be sent to the designated recipient's designated receiving method through the message sending configuration in the previously obtained alarm configuration information.

[0023] An alarm management device comprises: at least one memory and at least one processor;

[0024] The at least one memory is used to store a machine-readable program;

[0025] The at least one processor is used to call the machine-readable program to execute an alarm management method.

[0026] Compared with the prior art, the alarm management method and device of the present invention have the following outstanding beneficial effects:

[0027] The present invention collects, shields, silences and recovers the alarm data of different monitoring systems through a unified process and sends alarm notifications, integrates the alarm information of multiple monitoring systems, and realizes the centralized management and display of the alarm information of multiple monitoring components. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Attached Figure 1 The present invention is a workflow diagram of an alarm processing module in an alarm management method. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] A best embodiment is given below:

[0032] In an alarm management method of the present embodiment, first, the alarm control module guides the user to maintain alarm rules through the provided form page, including alarm rules, alarm shielding settings and alarm message template settings;

[0033] The alarm processing module processes and sends alarms in a continuous cycle according to the alarm rules. It adds restrictions to the alarm information according to the configured rules and then sends it to the monitoring system to obtain data. It also blocks, restores, silently judges and sends the returned data.

[0034] Among them, when the alarm rule is implemented in the alarm management module, the alarm rule sets the JavaScript expression. The user can set the expression to number>0, and the value of number is the number of data returned that meets the query conditions. The alarm will be issued only when the quantity condition meets the expression.

[0035] For example, in Prometheus, number indicates the number of metrics returned, which is equivalent to data.result.length. In Clickhouse, select count(*) is used to query the number of metrics that meet the conditions as number for judgment.

[0036] When setting alarm shielding, the user configures the alarm shielding label, specifies the alarm shielding label key as the shielding field key, the alarm shielding operator as the shielding judgment standard, and the alarm shielding value as the shielding field value. The shielding time can also be set. The alarm shielding settings are as follows:

[0037] key: http_status_code

[0038] operator:>

[0039] value: 400

[0040] You can also control the muting time, supporting periodic time muting and fixed time muting. The value of the periodic muting time type muteTimeType is periodicMutes, and you can set the week number, period start time, and stop time at the same time.

[0041] For example, if you set the week number to 0, the start time to 8:30, and the end time to 17:45, the alarm will be blocked from 8:30 to 17:45 every Monday, and no alarm information will be sent. If the value of the fixed time type muteTimeType is regularMutes, and you set the start time and end time at the same time, for example, the start time is 18:30 and the end time is 7:45, the alarm will be blocked from 18:30 to 7:45 the next day, and no alarm information will be sent. The settings are as follows:

[0042] Cycle time:

[0043] muteTimeType: periodicMutes

[0044] periodicMutes: [{"enableStime":"08:30", "enableEtime":"17:45", "enableDaysOfWeek":"0"},

[0045] {"enableStime":"05:15", "enableEtime":"18:00", "enableDaysOfWeek":"4,5"}]

[0046] Fixed time:

[0047] muteTimeType: regularMutes

[0048] beginTime:18:30

[0049] endTime:7:45

[0050] When setting the alarm message template, you can customize the message template according to the response data. When sending the actual alarm message, the fields will be replaced according to the alarm information and the customized message format will be sent. The Prometheus type message template is as follows:

[0051] <#list data.result as item>

[0052] ${item?index+1}. Warning information:

[0053] System: ${item.metric.project}

[0054] Tenant: ${item.metric.tenant}

[0055] Monitoring object: ${item.metric.agent_hostname}

[0056] Warning value: ${item.value[1]}%

[0057] < / #list> "

[0058] The Clickhouse type message template is as follows:

[0059] ${project} has ${NUMBER} abnormal logs in the last ${TIME_WINDOW} minutes. The most recent abnormal information:

[0060] Tenant: ${tenant}

[0061] Trace ID: ${traceid}

[0062] Log level: ${level}

[0063] Exception message: ${message}

[0064] like Figure 1 As shown, the workflow of the alarm processing module is described;

[0065] When adding restrictions to the data retrieval logic, the alarm shielding function is implemented for the Clickhouse data source type alarm by adding fields before sending the data retrieval logic to the data source.

[0066] Clickhouse type alarms need to first check whether the user has set the alarm shielding field. According to the alarm shielding label configured by the user, the label name is used as the field, the label operator is used as the judgment standard, and the label value is used as the shielding value. Among them, metricContract stores the alarm rules, and shieldInfo stores the shielding rules.

[0067] The original Clickhouse data retrieval logic statement is as follows:

[0068] select column1, column2 FROM table_name WHERE condition;

[0069] The actual SQL executed is as follows:

[0070] -- Query the number of data entries

[0071] select count(*) from (select column1, column2 FROM table_name WHEREcondition and shieldCondition1 and shieldCondition2 and created_at>=? andcreated_at <?)

[0072] -- Query the latest data

[0073] select * from table_name where 1=1 and created_at>=? and created_at <?order by created_at desc limit 1

[0074] When making alarm shielding judgments on the returned data, for Prometheus type alarms, after the data source returns the alarm information, the returned alarm data is filtered and controlled according to the shielding event tags in the current alarm configuration information to implement the alarm shielding function. Through jsonpath for tag matching, alarms containing shielded fields will not be sent.

[0075] According to the jsonpath shielding settings configured by the user, the returned alarm information is matched. The successfully matched alarm description contains a shielding field. By setting the shield field to true, it tells the subsequent processing that the alarm has been shielded and no notification is needed. Among them, shieldInfo stores the shielding rules, metricData stores the alarm data returned by the monitoring system, and alarmProcessLogger stores the alarm configuration information.

[0076] The format of monitoring data returned by Prometheus is as follows:

[0077] Prometheus returns data in the following format:

[0078] "status": "success",

[0079] "data":{

[0080] "resultType": "vector",

[0081] "result" : [

[0082] "metric" : {

[0083] "__name__" : "up",

[0084] "job" : "prometheus",

[0085] "instance" : "localhost:9090"

[0086] "value": [1435781451.781, "1"]

[0087] "metric" : {

[0088] "__name__" : "up",

[0089] "job" : "node",

[0090] "instance" : "localhost:9100"

[0091] "value" : [ 1435781451.781, "0" ]

[0092] According to the Prometheus return format, the jsonpath for label matching is as follows:

[0093] $.data.result[?(@.metric.conditionKey1==conditionValue1&&@.metric.conditionKey2==conditionValue2)].

[0094] When judging the alarm recovery of the returned data, for the Prometheus data source type alarm, for the alarm event that has already occurred, if you query it through promql again within the scheduling cycle and find that there is no returned content, it means that the current monitoring data no longer meets the threshold conditions specified in promql, indicating that the alarm has been restored.

[0095] The method for judging alarm recovery is to generate a unique HashCode based on the metric field value and the rule id, and compare the generated HashCode with the historical data of the current alarm in the alarm history information table. If the newly generated HashCode matches, it means that the alarm has not been restored. If the newly generated HashCode cannot be matched in the database, it means that the alarm is a newly generated alarm information. If there is a HashCode in the database and the current alarm does not generate a HashCode with the same value, it means that this alarm has been restored.

[0096] When making alarm silence judgment on the returned data, the alarm processing module matches the alarm data returned by the monitoring system according to the user-defined jsonpath silence condition, compares the current alarm silence times with the preset threshold of alarm silence according to the alarm silence setting and the record of alarm sending log, and calculates the time of the last alarm sending and the current time through the preset silence time. If the alarm in the alarm silence period is judged as an unrecovered alarm, the same alarm message will not be sent to the user again.

[0097] When sending alarms for returned data, alarm information that is not blocked and is not in a silent period will be converted into a message content format according to the message template configured by the user, and the alarm will be sent to the designated recipient's designated receiving method through the message sending configuration in the previously obtained alarm configuration information. The alarm information sent to the user after the format conversion according to the alarm message template is as follows:

[0098] 1. Warning information:

[0099] System: System 1

[0100] Tenant: Tenant 1

[0101] Host: 10.110.99.131

[0102] The memory usage is detected to be too high. The current memory usage is 82%.

[0103] Silence time: 60 minutes.

[0104] 2. Warning information:

[0105] System: System 2

[0106] Tenant: Tenant 2

[0107] Host: 10.110.92.129

[0108] The memory usage is detected to be too high. The current memory usage is 84%.

[0109] Silence time: 60 minutes.

[0110] Based on the above method, an alarm management device in this embodiment includes: at least one memory and at least one processor;

[0111] The at least one memory is used to store a machine-readable program;

[0112] The at least one processor is used to call the machine-readable program to execute an alarm management method.

[0113] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that comply with the claims of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0114] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An alarm management method, characterized in that: First, the alarm management and control module guides users to maintain alarm rules through the provided form page, including alarm rules, alarm shielding settings, and alarm message template settings; The alarm processing module processes and sends alarms in a continuous cycle according to the alarm rules. It adds restrictions to the alarm information according to the configured rules and then sends it to the monitoring system to obtain data. It also blocks, restores, silently judges and sends the returned data. The alarm rule sets a JavaScript expression. The user sets the expression to number > 0. The value of number is the number of returned data that meets the query conditions. The alarm will be triggered only when the number condition meets the expression. When setting alarm shielding, the user configures the alarm shielding label, specifies the alarm shielding label key as the shielding field key, the alarm shielding operator as the shielding judgment standard, the alarm shielding value as the shielding field value, and sets the shielding time to control the shielding time. It supports periodic time shielding and fixed time shielding. The value of the periodic shielding time type muteTimeType is periodicMutes, and the number of weeks, period start time and stop time are set at the same time. When setting the alarm message template, customize the message template according to the response data. When sending the actual alarm message, replace the fields according to the alarm information and send the customized message format. In the alarm processing module, when adding restrictions to the data acquisition logic, the alarm shielding is achieved by adding fields before sending the data acquisition logic to the data source for the Clickhouse data source type alarm; First, check whether the user has set the alarm shielding field. According to the alarm shielding tag configured by the user, use the tag name as the field, the tag operator as the judgment standard, and the tag value as the shielding value; When making alarm shielding judgment on the returned data, for Prometheus type alarms, after the data source returns the alarm information, the returned alarm data is filtered and controlled according to the shielding event label in the current alarm configuration information to achieve alarm shielding. The label is matched through jsonpath, and the alarm containing the shielded field will not be sent; When judging the alarm recovery of the returned data, for the Prometheus data source type alarm, for the alarm event that has been generated, if you query it through promql again within the scheduling cycle and find that there is no returned content, it means that the current monitoring data no longer meets the threshold conditions specified in promql, indicating that the alarm has been restored; The method for judging whether the alarm has been restored is to generate a unique HashCode based on the metric field value and the rule id, and compare the generated HashCode with the existing historical data of the current alarm in the alarm history information table. If the newly generated HashCode matches, it means that the alarm has not been restored; If the newly generated HashCode does not match in the database, it means that the alarm is a newly generated alarm information. If the HashCode already exists in the database and the current alarm does not generate the same HashCode value, it means that the alarm has been restored.

2. An alarm management method according to claim 1, characterized in that: When making alarm silence judgment on the returned data, the alarm processing module matches the alarm data returned by the monitoring system according to the user-defined jsonpath silence condition, compares the current alarm silence times with the preset threshold of alarm silence according to the alarm silence setting and the record of alarm sending log, and calculates the time of the last alarm sending and the current time through the preset silence time. If the alarm in the alarm silence period is judged as an unrecovered alarm, the same alarm message will not be sent to the user again.

3. An alarm management method according to claim 2, characterized in that: When sending an alarm for the returned data, the alarm information that is not blocked and is not in a silent period will be converted into a message content format according to the message template configured by the user, and the alarm will be sent to the designated receiving method of the designated recipient through the message sending configuration in the previously obtained alarm configuration information.

4. An alarm management device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 3.

Images