A monitoring alarm method and apparatus

Abstract

The application discloses a kind of monitoring alarm method and device, the method comprises the following steps: by message queue, the service data reported by each business service is collected, and the statistical metadata in unit time is calculated according to the service data;Based on the statistical metadata, the statistical result of multiple monitoring indexes in multiple unit time dimensions is statistically in real time, and the multiple monitoring indexes include request total number, request failure total number, failure average response duration and success average response duration;According to the statistical result, it is determined whether to send alarm notification.The application embodiment determines whether to send alarm notification according to the statistical result of multiple monitoring indexes in multiple unit time dimensions statistically in real time, can reduce the data amount of calculation of monitoring system, avoid the occurrence of system false alarm, improve the credibility of statistical monitoring early warning service.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a monitoring and alarm method and device. Background Technology

[0002] In the production and operation of enterprise SaaS (Software-as-a-Service) platforms, the primary responsibility is to ensure high concurrency, high performance, and high availability of business capabilities. However, a mature SaaS platform architecture is often very complex, resulting in complex and extensive maintenance work. If a SaaS platform experiences a sudden failure and cannot provide normal business services, in addition to economic losses, the greater loss is the damage to the SaaS platform and the company's reputation. Therefore, statistical monitoring and early warning systems, including but not limited to SaaS platform systems, are essential. Statistical data can serve as a data tool for business operations, provide alert capabilities for online operations, and enable early warning systems to detect, analyze, and resolve production problems as early as possible.

[0003] Currently, SaaS platform monitoring and early warning systems on the market can provide alarms for simple business systems, but they require a large amount of data to be calculated, making the work complex and prone to errors.

[0004] Application content

[0005] The purpose of this application is to provide a monitoring and alarm method and apparatus to address the shortcomings of existing monitoring and early warning systems that require a large amount of computational data.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows:

[0007] Firstly, a monitoring and alarm method is provided, including the following steps:

[0008] The system collects business data reported by various business services through a message queue and calculates statistical metadata per unit time based on the business data.

[0009] Based on the statistical metadata, the statistical results of multiple monitoring indicators within multiple unit time dimensions are statistically analyzed in real time. The multiple monitoring indicators include the total number of requests, the total number of failed requests, the average response time for failures, and the average response time for successes.

[0010] Based on the statistical results, determine whether to send an alarm notification.

[0011] Secondly, a monitoring and alarm device is provided, comprising:

[0012] The calculation module is used to collect business data reported by various business services through a message queue, and to calculate statistical metadata per unit time based on the business data.

[0013] The statistics module is used to perform real-time statistics on multiple monitoring metrics within multiple time dimensions based on the statistical metadata. The multiple monitoring metrics include the total number of requests, the total number of failed requests, the average response time for failures, and the average response time for successes.

[0014] The determination module is used to determine whether to send an alarm notification based on the statistical results.

[0015] This application embodiment determines whether to send an alarm notification based on the statistical results of multiple monitoring indicators within multiple unit time dimensions in real time, which can reduce the amount of computational data of the monitoring system, avoid false alarms, and improve the reliability of statistical monitoring and early warning services. Attached Figure Description

[0016] Figure 1 This is a flowchart of a monitoring and alarm method provided in an embodiment of this application;

[0017] Figure 2 This is a schematic diagram of the structure of a monitoring and alarm device provided in an embodiment of this application;

[0018] Figure 3 This is a schematic diagram of the average value of the calculated index provided in the embodiments of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] When business services experience operational anomalies or malfunctions due to program bugs, third-party dependencies, or malicious attacks, SaaS platforms typically employ corresponding statistical monitoring and alarm systems to assist operations. Alarms can be generated by comparing fixed window data with pre-configured business thresholds, providing simple alarm functionality for business systems. However, this approach presents several challenges: 1. It cannot dynamically adjust fixed thresholds. For example, if service request volume experiences a surge at a certain moment (e.g., client-scheduled API calls), resulting in peak or trough system pressure, the request statistics within a given time window will likely exceed the pre-configured threshold. Issuing alarms in such cases would disrupt the normal work of operations and development teams, leading to false alarms and reducing the reliability of the monitoring system. 2. The amount of data required for computation is substantial.

[0021] To address the above issues, this application proposes a complete statistical, monitoring, and alarm system. It generates monitoring and alarm data by calculating the average business value across various time windows, rather than the absolute value of a single unit window. Secondary analysis and statistics are performed based on the statistical service data. The monitoring capability API is scheduled periodically, analyzing the basic statistical data in real time and calculating according to existing monitoring rules. When a monitoring indicator exceeds a configured threshold, the top N factors contributing to the alarm are generated, triggering the alarm system to send alarm information to individuals or groups. Notification methods are not limited to email, SMS, and office IM systems. This reduces false alarms, improves the reliability of statistical monitoring and early warning services, and enables rapid detection and resolution of faults in case of anomalies, preventing enterprise losses.

[0022] The monitoring and alarm method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0023] like Figure 1 The diagram shown is a flowchart of a monitoring and alarm method provided in an embodiment of this application. The method includes the following steps:

[0024] Step 101: Collect business data reported by each business service through a message queue, and calculate statistical metadata per unit time based on the business data.

[0025] Step 102: Based on the statistical metadata, perform real-time statistical analysis of the statistical results of multiple monitoring indicators within multiple unit time dimensions.

[0026] The monitoring metrics include total number of requests, total number of failed requests, average response time for failed requests, and average response time for successful requests. The time unit dimensions include first window time and second window time, where the first window time is shorter than the second window time, and the first window time and the second window time have the same right window boundary and different left window boundaries.

[0027] Step 103: Based on the statistical results, determine whether to send an alarm notification.

[0028] Specifically, if the ratio of the average failure response time within the first window period to the average failure response time within the second window period is greater than a preset threshold, it is determined that an alarm notification needs to be sent; if the ratio of the average failure response time within the first window period to the average failure response time within the second window period is less than the preset threshold, it is determined that an alarm notification does not need to be sent.

[0029] In this embodiment, the weighted trend value of each monitoring indicator can be calculated based on the changing trend of the statistical results of each monitoring indicator within each unit time dimension and the weight of each monitoring indicator; if the weighted trend value is greater than a preset threshold, it is determined that an alarm notification needs to be sent.

[0030] In addition, the fault value of each interface can be calculated based on the statistical results of multiple monitoring indicators corresponding to each interface and the weight of each monitoring indicator; the fault values ​​of each interface can be sorted, and the interface that has failed can be determined based on the sorting results.

[0031] This application embodiment determines whether to send an alarm notification based on the statistical results of multiple monitoring indicators within multiple unit time dimensions in real time, which can reduce the amount of computational data of the monitoring system, avoid false alarms, and improve the reliability of statistical monitoring and early warning services.

[0032] In this embodiment, the system built on a microservice architecture largely relies on HTTP protocol interfaces to transmit data. Each interface has metrics such as total number of requests, number of failed requests, number of successful requests, and response time. This multi-business dimension data can serve as the basic metadata for statistical alarms. Secondary analysis and statistics are performed on this source data to trigger alarm data.

[0033] The statistical monitoring and alarm system in this embodiment consists of a statistical system, a monitoring system, and an alarm system. The statistical system is built based on technologies such as log collection, message queues, and databases. The monitoring and alarm systems are triggered by timed scheduling. Based on the statistical summary metadata within a unit of time, they perform secondary analysis and real-time calculations to determine whether an alarm is needed, the alarm information, and the factors that caused the alarm. At the same time, the alarm is recorded, which can depict the historical number and frequency of alarms.

[0034] Specifically, the implementation methods of the business statistics monitoring and alarm system include:

[0035] 1. Build a statistical service: The message queue collects business data reported by various business services and summarizes statistical metadata within a unit of time through streaming computing.

[0036] 2. Establish a monitoring and alarm service: The monitoring and alarm service provides an interface for scheduled services to call. This interface provides real-time statistics on various monitoring metrics (such as total number of requests, total number of failed requests, average failure time, and average success time) across multiple time units (1m / 5m / 10 / 20m / 40m). It also determines whether an alarm is needed and the alarm message content based on thresholds. Furthermore, to address the potential for false alarms during long alarm windows, an optimized algorithm can be implemented to perform reverse calculations and determine if the fault is showing signs of recovery. This can suppress alarms during long time windows, making the alarm system more accurate and increasing its reliability.

[0037] 3. Set up a timed scheduling and triggering service: periodically call the alarm capability interface of the monitoring and alarm service, and at the same time determine whether it is really necessary to send an alarm notification to the person in charge or the responsible organization based on the data returned by the alarm interface.

[0038] 4. Calculate the various causes of this alarm based on the diagnostic algorithm.

[0039] The diagnostic algorithm can be based on the ratio of the average value within a small unit window to the average value within a large unit window. Assuming the small unit window time is w, the large unit window time is W, and the business metric is average response time (RT), when w = 1 minute and W = 5 minutes, a w:W ratio of 3.57 (a pre-set threshold) indicates that the average response time for failed interface calls within 1 minute of this business metric has increased by 3.57 times compared to the average response time within 5 minutes. This also approximates an increase of 10 times compared to the average response time within the previous minute. If the window ratio is greater than or equal to the pre-set threshold, a system failure has occurred, and an alarm should be generated. For example, at any given time, the average values ​​for a given metric over the most recent 1 minute, 5 minutes, 10 minutes, and 20 minutes can be calculated, i.e., avg1, avg5, avg10, and avg20. Figure 3 As shown, this application's embodiments do not compare the absolute values ​​of each statistical indicator with the threshold, but rather use the ratio of statistical indicator values ​​from adjacent time dimensions. Therefore, it has two prominent features: using relative ratios instead of absolute values, and using indicator values ​​from different time dimensions with the same right boundary but different left boundaries. The threshold setting can be based on different application scenarios and indicator characteristics, designing different threshold lookup tables. These can be divided into increasing and decreasing indicator values, and the threshold can be selected according to different requirements for alarm detection sensitivity. The table below is only for illustration.

[0040] Threshold lookup example - example of increasing indicator value (a is a quantity value):

[0041]

[0042] Threshold lookup example - Example of indicator value decrease (a is a quantity value for illustration):

[0043]

[0044]

[0045] Furthermore, based on the weight values ​​of each business monitoring metric and the top N interfaces for each metric, the fault weight value (referred to as the fault value) of each interface for each business monitoring metric failure can be calculated, thereby analyzing the specific interface that caused the failure. For example, for the business metrics: the weight of Total Failed Response Time (FRT) is 0.6, the weight of Average Response Time for Failed Requests (AVGRT) is 0.2, and the weight of the number of failed requests is 0.2; based on the top 20 interfaces with the highest total RT for failed API requests...

[0046] The table contains statistical metadata for the top 20 lists of APIs with the highest average response time (RT) and the top 20 lists of APIs with the highest number of failed requests. It can calculate the failure value for each of the top 20 lists. For example, if API A exists in all three failure lists, its failure value is: 20 (FRT score) * 0.6 (FRT weight) + 18 (AVGRT score) * 0.2 (AVGRT weight) + 18 (request failure count score) * 0.2 (failure weight) = 19.2. If API X exists only in some failure lists, only its corresponding failure value needs to be calculated. Finally, by sorting the failure values ​​of each API, the Top N list of APIs causing the failure can be obtained. Developers can use this list to quickly locate, troubleshoot, and resolve online failures.

[0047] The sum of the weights of all metrics should equal 1. The higher the ranking of the interface, the higher the score should be. The score can be configured through a configuration file to facilitate subsequent business adjustments.

[0048]

[0049]

[0050]

[0051]

[0052] Therefore, the monitoring and alarm granularity of this application embodiment is controllable, and the statistical dimensions are controllable; the top N alarm causes are classified and sorted for quick problem location; and historical alarm records are available at a glance. Traditional technologies use the absolute value of an indicator describing the system's operational status to set alarm thresholds, often encountering difficulties in continuous operation due to inaccurate threshold settings or the need for frequent adjustments. This application embodiment, however, does not require absolute value threshold settings for system indicators, but instead adopts an operationally friendly ratio threshold setting method that adapts to the operating conditions, thus avoiding the operational dilemma of frequent adjustments to absolute value thresholds. Compared with existing anomaly detection technologies, this application embodiment has the ability to simultaneously examine system trend changes across multiple time windows, ensuring neither false alarms nor missed alarms. Looking at a 1-minute time window, the trend change does not meet the alarm conditions; looking at a 5-minute time window, the trend change does meet the alarm conditions. This application embodiment can not only perform trend alarms but also cause diagnosis, identifying the individual main causes of global trend alarms, making alarms more accurate and facilitating rapid fault handling by R&D personnel.

[0053] Furthermore, the embodiments of this application have the advantage of low computational complexity. Existing technologies generally use a year-on-year comparison method; for example, comparing indicator values ​​across three time windows requires calculating at least six indicator values ​​for the calculation. However, the embodiments of this application can complete the comparison of indicator values ​​across three time windows using only four indicator values.

[0054] Specifically, existing month-on-month calculation methods require six operands to calculate the ratio: the average value of the current 1-minute indicator / the average value of the previous 1-minute indicator, the average value of the current 5-minute indicator / the average value of the previous 5-minute indicator, and the average value of the current 10-minute indicator / the average value of the previous 10-minute indicator. In contrast, the embodiments of this application only require four operands to calculate the ratio: the average value of the current 1-minute indicator / the average value of the current 5-minute indicator, the average value of the current 5-minute indicator / the average value of the current 10-minute indicator, and the average value of the current 10-minute indicator / the average value of the current 20-minute indicator.

[0055] like Figure 2 As shown, this is a monitoring and alarm device according to an embodiment of the present application, comprising:

[0056] The calculation module 210 is used to collect business data reported by various business services through a message queue, and to calculate statistical metadata per unit time based on the business data.

[0057] The statistics module 220 is used to perform real-time statistics on multiple monitoring indicators within multiple unit time dimensions based on the statistical metadata.

[0058] The monitoring metrics include total number of requests, total number of failed requests, average response time for failed requests, and average response time for successful requests. The time unit dimensions include first window time and second window time, where the first window time is shorter than the second window time, and the first window time and the second window time have the same right window boundary and different left window boundaries.

[0059] The determination module 230 is used to determine whether to send an alarm notification based on the statistical results.

[0060] Specifically, the determining module 230 is used to determine that an alarm notification needs to be sent when the ratio of the average failure response time within the first window time to the average failure response time within the second window time is greater than a preset threshold; and to determine that an alarm notification does not need to be sent when the ratio of the average failure response time within the first window time to the average failure response time within the second window time is less than the preset threshold.

[0061] In this embodiment, the determining module 230 can also be specifically used to calculate the weighted trend value of each monitoring indicator based on the changing trend of the statistical results of each monitoring indicator within each unit time dimension and the weight of each monitoring indicator; and to determine that an alarm notification needs to be sent if the weighted trend value is greater than a preset threshold.

[0062] Furthermore, the aforementioned device also includes:

[0063] The sorting module is used to calculate the fault value of each interface based on the statistical results of multiple monitoring indicators corresponding to each interface and the weight of each monitoring indicator; sort the fault values ​​of each interface; and determine the interface that has failed based on the sorting results.

[0064] This application embodiment determines whether to send an alarm notification based on the statistical results of multiple monitoring indicators within multiple unit time dimensions in real time, which can reduce the amount of computational data of the monitoring system, avoid false alarms, and improve the reliability of statistical monitoring and early warning services.

[0065] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0066] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0067] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A monitoring and alarm method, characterized in that, Includes the following steps: The system collects business data reported by various business services through a message queue and calculates statistical metadata per unit time based on the business data. Based on the statistical metadata, the statistical results of multiple monitoring indicators within multiple unit time dimensions are statistically analyzed in real time. The multiple monitoring indicators include the total number of requests, the total number of failed requests, the average response time for failures, and the average response time for successes. Based on the statistical results, determine whether to send an alarm notification; The multiple unit time dimensions include a first window time and a second window time, wherein the first window time is less than the second window time, and the first window time and the second window time have the same right window boundary and different left window boundaries; The step of determining whether to send an alarm notification based on the statistical results specifically includes: If the ratio of the average failure response time within the first window period to the average failure response time within the second window period is greater than a preset threshold, it is determined that an alarm notification needs to be sent.

2. The method according to claim 1, characterized in that, The step of determining whether to send an alarm notification based on the statistical results specifically includes: If the ratio of the average failure response time within the first window period to the average failure response time within the second window period is less than a preset threshold, it is determined that no alarm notification needs to be sent.

3. The method according to claim 1, characterized in that, The step of determining whether to send an alarm notification based on the statistical results specifically includes: Based on the changing trends of the statistical results of each monitoring indicator within each unit time dimension, and the weight of each monitoring indicator, calculate the weighted trend value of each monitoring indicator. If the weighted trend value is greater than a preset threshold, it is determined that an alarm notification needs to be sent.

4. The method according to claim 1, characterized in that, Also includes: Based on the statistical results of multiple monitoring metrics corresponding to each interface and the weight of each monitoring metric, the fault value of each interface is calculated. The fault values ​​of each interface are sorted, and the interface that has failed is determined based on the sorting results.

5. A monitoring and alarm device, characterized in that, include: The calculation module is used to collect business data reported by various business services through a message queue, and to calculate statistical metadata per unit time based on the business data. The statistics module is used to perform real-time statistics on multiple monitoring metrics within multiple time dimensions based on the statistical metadata. The multiple monitoring metrics include the total number of requests, the total number of failed requests, the average response time for failures, and the average response time for successes. The determination module is used to determine whether to send an alarm notification based on the statistical results. The multiple unit time dimensions include a first window time and a second window time, wherein the first window time is less than the second window time, and the first window time and the second window time have the same right window boundary and different left window boundaries; The determining module is specifically used to determine that an alarm notification needs to be sent when the ratio of the average failure response time within the first window time to the average failure response time within the second window time is greater than a preset threshold.

6. The apparatus according to claim 5, characterized in that, The determining module is specifically used to determine that no alarm notification needs to be sent when the ratio of the average failure response time within the first window time to the average failure response time within the second window time is less than a preset threshold.

7. The apparatus according to claim 5, characterized in that, The determining module is specifically used to calculate the weighted trend value of each monitoring indicator based on the changing trend of the statistical results of each monitoring indicator within each unit time dimension and the weight of each monitoring indicator. If the weighted trend value is greater than a preset threshold, it is determined that an alarm notification needs to be sent.

8. The apparatus according to claim 5, characterized in that, Also includes: The sorting module is used to calculate the fault value of each interface based on the statistical results of multiple monitoring indicators corresponding to each interface and the weight of each monitoring indicator. The fault values ​​of each interface are sorted, and the interface that has failed is determined based on the sorting results.

Images