Elastic monitoring alarm method and system based on multistage cooperative verification

Abstract

The invention provides an elastic monitoring alarm method and system based on multistage cooperative verification. The method comprises the step of remarkably improving the alarm accuracy through a three-level verification mechanism. The first-stage verification adopts intelligent anomaly detection, compares a current index with a historical baseline, and identifies a preliminary anomaly; in the second-stage verification, association index analysis is introduced, and pseudo anomalies caused by normal business fluctuation are filtered; and the third-stage verification performs cross-customer collaborative analysis to distinguish individual anomaly from global events. In addition, fault root causes are automatically positioned through dependency link analysis, and structured alarm information containing causes, confidence coefficients and processing suggestions is generated. And a closed-loop learning mechanism is established, operation and maintenance feedback is continuously collected, and alarm parameters are automatically adjusted by adopting Bayesian optimization. According to the method, the false alarm rate is reduced, the root cause positioning accuracy is improved, the average repair time is shortened, and the IT operation and maintenance efficiency is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a flexible monitoring and alarm method and system based on multi-level collaborative verification, which is applied in IT system monitoring and management scenarios to achieve high-precision, low-false-alarm system monitoring and alarm functions. Background Technology

[0002] With the widespread adoption of cloud computing and distributed systems, IT system monitoring and alarm mechanisms have become increasingly important. Currently, most monitoring and alarm systems primarily use static thresholds or simple comparisons with historical data to determine system anomalies. Specifically, when a monitoring indicator exceeds a preset threshold or deviates significantly from historical data, the system triggers an alarm mechanism.

[0003] However, existing monitoring and alarm methods have several technical limitations: First, fixed thresholds cannot adapt to normal business fluctuations (such as traffic surges during marketing campaigns), leading to a large number of false alarms; second, the anomaly verification methods of traditional systems are often too crude, usually requiring several consecutive triggers before an alarm is confirmed, or relying entirely on manual judgment by operations and maintenance personnel, resulting in response delays and inaccurate verification; third, existing alarm systems usually lack the ability to automatically analyze the root cause of anomalies, only able to point out abnormal phenomena but unable to quickly locate the fault point, prolonging the troubleshooting time; finally, traditional systems lack sufficient intelligence, with threshold adjustments and verification rules heavily reliant on human experience, making them unable to adapt to changes in business.

[0004] These technical issues result in a high false alarm rate and low alarm response and processing efficiency in existing monitoring and alarm systems, making it difficult to meet the needs of modern IT operations and maintenance for efficient and accurate monitoring. Summary of the Invention

[0005] The purpose of this invention is to provide a flexible monitoring and alarm method and system based on multi-level collaborative verification, which aims to solve the technical problems of existing monitoring and alarm systems, such as high false alarm rate, lagging and crude verification, lack of root cause localization, and insufficient system intelligence.

[0006] To achieve the above objectives, the present invention provides a resilient monitoring and alarm method based on multi-level collaborative verification, comprising:

[0007] Obtain historical monitoring indicator data corresponding to real-time monitoring indicator data, verify the historical monitoring indicator data, and obtain verified historical monitoring indicator data.

[0008] Based on the verified historical monitoring indicator data, a baseline value is calculated, and the deviation between the current monitoring indicator value in the real-time monitoring indicator data and the baseline value is calculated. It is then determined whether the deviation is within the tolerance range to obtain the first-level verification result.

[0009] When the deviation indicated by the first-level verification result exceeds the tolerance range, the associated indicator data related to the real-time monitoring indicator data is obtained, the expected value of the real-time monitoring indicator data is calculated based on the associated indicator data, and the current monitoring indicator value is compared with the expected value to obtain the second-level verification result.

[0010] When the second-level verification result indicates that the change in the current monitoring indicator value cannot be explained by the change in the associated indicator data, obtain the same indicator data of the similar object group, compare the monitoring indicator distribution of the abnormal object with the monitoring indicator distribution of the similar object group, calculate the statistical test p value, and obtain the third-level verification result.

[0011] When the p-value in the third-level verification result is less than the preset significance level threshold, the upstream dependent service list is obtained based on the dependency relationship, the health status of the upstream dependent service is queried, and the root cause localization result is obtained; when the p-value in the third-level verification result is greater than or equal to the preset significance level threshold, it is determined to be a group anomaly, confirmed as a global event alarm, and alarm information is directly generated.

[0012] Based on the root cause localization results, an alarm message is generated, the alarm message is pushed, and feedback annotations on the alarm message are received to obtain feedback data.

[0013] The false alarm rate is calculated based on the feedback data. When the false alarm rate exceeds a preset threshold, the alarm decision parameters are adjusted to obtain an optimized alarm decision model.

[0014] Further, the step of verifying the historical monitoring indicator data to obtain verified historical monitoring indicator data includes:

[0015] Acquire historical monitoring indicator data within a preset time window, wherein the historical monitoring indicator data is a sequence of historical monitoring indicators arranged in chronological order;

[0016] Calculate the mean and standard deviation of the historical monitoring indicator sequence data, and determine the 3-Sigma threshold range based on the mean and standard deviation;

[0017] Traverse each data point in the historical monitoring indicator sequence data, determine whether the value of each data point falls outside the 3-Sigma threshold range, and mark the data points that fall outside the 3-Sigma threshold range as historical anomalies;

[0018] The historical anomalies are supplemented using linear interpolation or spline interpolation methods to obtain verified historical monitoring index data.

[0019] Further, the step of calculating a baseline value based on the verified historical monitoring indicator data, calculating the deviation between the current monitoring indicator value in the real-time monitoring indicator data and the baseline value, determining whether the deviation is within the tolerance range, and obtaining the first-level verification result includes:

[0020] Based on the timestamp of the real-time monitoring indicator data, obtain the historical monitoring indicator data of the same period with the corresponding time granularity from the verified historical monitoring indicator data, and calculate the historical average and historical fluctuation range of the historical monitoring indicator data.

[0021] Obtain business context metadata, which includes marketing campaign intensity weight, seasonality index, and resource coefficient; and multiply the marketing campaign intensity weight, the seasonality index, and the resource coefficient to calculate the context coefficient;

[0022] The baseline value is obtained by multiplying the historical average value by the context coefficient.

[0023] Multiplying the historical fluctuation range by the context coefficient yields the dynamic tolerance range;

[0024] The deviation is obtained by calculating the absolute value of the difference between the current monitoring indicator value in the real-time monitoring indicator data and the baseline value;

[0025] Determine whether the deviation is less than or equal to the dynamic tolerance range. When the deviation is less than or equal to the dynamic tolerance range, it is determined to be a normal fluctuation. When the deviation is greater than the dynamic tolerance range, it is marked as a suspected anomaly and carries the baseline value, the deviation, and the context coefficient to obtain the first-level verification result.

[0026] Further, the step of acquiring associated indicator data related to the real-time monitoring indicator data, calculating the expected value of the real-time monitoring indicator data based on the associated indicator data, comparing the current monitoring indicator value with the expected value, and obtaining the second-level verification result includes:

[0027] Obtain a pre-configured association indicator mapping table, and determine the association indicator data that are associated with the real-time monitoring indicator data based on the association indicator mapping table;

[0028] Historical data of the related indicator data and historical data of the real-time monitoring indicator data are obtained. A linear regression algorithm is used to fit the relationship between the historical data of the related indicator data and the historical data of the real-time monitoring indicator data to obtain the regression coefficients and intercept parameters, and a regression model is constructed.

[0029] Obtain the current actual value of the associated indicator data, substitute the current actual value of the associated indicator data into the regression model, and calculate the expected value of the real-time monitoring indicator data.

[0030] Calculate the relative error between the current monitoring indicator value and the expected value, and determine whether the relative error is less than a preset error threshold.

[0031] When the relative error is less than the preset error threshold, it is determined that the change in the current monitoring indicator value can be explained by the change in the associated indicator data, and an alarm is filtered out. When the relative error is greater than or equal to the preset error threshold, it is determined to be a real anomaly and carries the expected value and the relative error to obtain the second-level verification result.

[0032] Further, the step of obtaining the same indicator data of similar object groups, comparing the monitoring indicator distribution of abnormal objects with the monitoring indicator distribution of the similar object groups, calculating the statistical test p-value, and obtaining the third-level verification result includes:

[0033] Construct customer feature vectors that include industry type, business scale, functional preferences and geographical distribution, and normalize the customer feature vectors of all customers to obtain normalized customer feature vectors.

[0034] The locality-sensitive hashing algorithm is used to perform fast approximate clustering on the normalized customer feature vectors, and the clustering results are updated offline at preset time intervals to obtain customer clusters.

[0035] Assign a similar customer cluster ID to each customer in the customer cluster, and cache the list of member customers corresponding to the similar customer cluster ID;

[0036] Based on the real-time monitoring indicator data that are determined to be real anomalies in the second-level verification results, the corresponding anomaly object identifier is extracted, the similar customer cluster ID of the customer corresponding to the anomaly object identifier is determined, and other customers with the same similar customer cluster ID are obtained from the member customer list to obtain the similar object group.

[0037] Obtain the same indicator data of the similar object group within the same time period, and construct the monitoring indicator distribution of the similar object group;

[0038] Based on the real-time monitoring indicator data that are determined to be real anomalies in the second-level verification results, the monitoring indicator distribution of the anomaly object is constructed.

[0039] The Mann-Whitney U test method is used to compare the distribution of monitoring indicators of the abnormal object with the distribution of monitoring indicators of the similar object group, and the p-value is calculated.

[0040] Determine whether the p-value is less than a preset significance level threshold. When the p-value is greater than or equal to the preset significance level threshold, determine that the abnormal object and the group behavior are consistent and confirm it as a global event alarm. When the p-value is less than the preset significance level threshold, determine that it is an individual abnormality and carry the p-value and the monitoring index distribution of the similar object group to obtain the third-level verification result.

[0041] Furthermore, the step of obtaining the list of upstream dependent services based on dependency relationships, querying the health status of the upstream dependent services, and obtaining the root cause localization result includes:

[0042] Obtain business log data containing TraceID, analyze the business log data to mine business processes, extract service call relationships, and construct a business link dependency graph; wherein the business link dependency graph describes the call dependency relationship between services and applications.

[0043] Based on the real-time monitoring indicator data that are determined to be individual anomalies in the third-level verification results, the corresponding anomaly object identifier and anomaly service identifier are extracted.

[0044] Based on the business link dependency graph, query all upstream dependent services of the current abnormal service corresponding to the abnormal service identifier to obtain the list of upstream dependent services;

[0045] Parallel query of monitoring metric data of each upstream dependent service in the upstream dependent service list, determine whether each upstream dependent service in the upstream dependent service list has triggered an abnormal alarm, and obtain the health status of each upstream dependent service in the upstream dependent service list;

[0046] A rule-based reasoning engine is used to traverse the list of upstream dependent services. When the health status of any upstream dependent service is abnormal, the upstream dependent service is marked as the root cause service. When the health status of all upstream dependent services in the list is normal, the root cause is located as an internal or unmonitored third-party dependency of this service, and the root cause location result is obtained.

[0047] Furthermore, the step of generating alarm information based on the root cause localization result and pushing the alarm information includes:

[0048] Based on the root cause localization results, the root cause service identifier, the abnormal object identifier, and the abnormal service identifier are extracted.

[0049] Extract the names of abnormal indicators from the real-time monitoring indicator data;

[0050] Extract the deviation from the first-level verification results;

[0051] Extract the relative error from the second-level verification results;

[0052] Extract the p-value from the third-level verification result;

[0053] The root cause service name is determined based on the root cause service identifier. When the root cause location result indicates that the root cause is located within this service, the service name corresponding to the abnormal service identifier is used as the root cause service name. When the root cause location result indicates that the root cause is located in an upstream dependent service, the name of the upstream dependent service marked as the root cause service is used as the root cause service name.

[0054] The verification confidence level is calculated based on the deviation, the relative error, and the p-value; wherein the verification confidence level is a weighted comprehensive score of the three-level verification results.

[0055] Based on the root cause service name and the verification confidence level, processing suggestions are generated using preset processing suggestion generation rules; wherein the processing suggestions include suggested investigation directions and suggested processing measures;

[0056] Construct an alarm information data structure, which includes the alarm timestamp, the abnormal object identifier, the abnormal service name corresponding to the abnormal service identifier, the abnormal indicator name, the deviation, the root cause service name, the verification confidence level, and the processing suggestion;

[0057] The alarm information data structure is encapsulated in JSON format or message queue format and pushed to the operation and maintenance platform through API interface or message push service.

[0058] Furthermore, the step of receiving feedback annotations for the alarm information to obtain feedback data includes:

[0059] The operation and maintenance platform receives processing operations from operation and maintenance personnel on the alarm information data structure, including confirming it as a real anomaly or marking it as a false alarm.

[0060] The processing operation is stored as a feedback label in the feedback database, and the feedback data is obtained by associating it with the unique identifier of the alarm information data structure and the alarm rule identifier.

[0061] Further, the step of calculating the false alarm rate based on the feedback data, and adjusting the alarm decision parameters when the false alarm rate exceeds a preset threshold to obtain an optimized alarm decision model, includes:

[0062] Periodically scan the preset alarm rule set and extract the feedback label corresponding to each alarm rule in the alarm rule set within the preset time period from the feedback database;

[0063] Based on the feedback annotation, the total number of alarms triggered by each alarm rule in the alarm rule set within the preset time period and the number of alarms marked as false alarms are counted.

[0064] Based on the total number of alarms and the number of alarms marked as false alarms, calculate the false alarm rate for each alarm rule in the alarm rule set; wherein the false alarm rate is the ratio of the number of alarms marked as false alarms to the total number of alarms.

[0065] Determine whether the false alarm rate exceeds a preset false alarm rate threshold, filter out alarm rules whose false alarm rate exceeds the preset false alarm rate threshold, and obtain a set of alarm rules to be optimized;

[0066] For each alarm rule to be optimized in the set of alarm rules to be optimized, a parameter search space is defined based on each alarm rule to be optimized; wherein the parameter search space includes the historical window length, tolerance factor, and weight parameters of each verification level in the baseline calculation;

[0067] A Bayesian optimization algorithm is used to construct a Gaussian process surrogate model between parameter combinations and the false alarm rate based on the parameter search space. Candidate parameter combinations are then iteratively selected in the parameter search space using a data acquisition function.

[0068] The historical monitoring indicator data and corresponding feedback labels that have triggered alarms within the preset time period are obtained from the feedback database to obtain the verification historical dataset.

[0069] For each candidate parameter combination in the candidate parameter combinations, each candidate parameter combination is applied to each alarm rule to be optimized. The alarm triggering process is simulated based on the verification historical dataset, the simulated false alarm rate is calculated, and the Gaussian process proxy model is updated based on the simulated false alarm rate to obtain the updated Gaussian process proxy model.

[0070] Based on the updated Gaussian process proxy model, after a preset number of iterations, the candidate parameter combination with the lowest simulation false alarm rate is selected as the optimal parameter.

[0071] The optimal parameters are hot-updated to each alarm rule to be optimized in the online alarm decision engine through the configuration management system to obtain the optimized alarm decision model.

[0072] This invention also provides a resilient monitoring and alarm system based on multi-level collaborative verification, comprising:

[0073] The data acquisition and verification module is used to acquire historical monitoring indicator data corresponding to real-time monitoring indicator data, verify the historical monitoring indicator data, and obtain verified historical monitoring indicator data.

[0074] The first-level verification module is used to calculate the baseline value based on the verified historical monitoring indicator data, calculate the deviation between the current monitoring indicator value in the real-time monitoring indicator data and the baseline value, determine whether the deviation is within the tolerance range, and obtain the first-level verification result.

[0075] The second-level verification module is used to obtain associated indicator data related to the real-time monitoring indicator data when the deviation indicated by the first-level verification result exceeds the tolerance range, calculate the expected value of the real-time monitoring indicator data based on the associated indicator data, compare the current monitoring indicator value with the expected value, and obtain the second-level verification result.

[0076] The third-level verification module is used to obtain the same indicator data of similar object groups when the second-level verification result indicates that the change of the current monitoring indicator value cannot be explained by the change of the associated indicator data, compare the monitoring indicator distribution of the abnormal object with the monitoring indicator distribution of the similar object group, calculate the statistical test p value, and obtain the third-level verification result.

[0077] The root cause localization module is used to obtain the upstream dependent service list based on the dependency relationship and query the health status of the upstream dependent services to obtain the root cause localization result when the p value in the third-level verification result is less than the preset significance level threshold; when the p value in the third-level verification result is greater than or equal to the preset significance level threshold, it is determined to be a group anomaly, confirmed as a global event alarm and directly generates alarm information.

[0078] The alarm push and feedback module is used to generate alarm information based on the root cause localization results, push the alarm information, and receive feedback annotations on the alarm information to obtain feedback data.

[0079] The self-optimization module is used to calculate the false alarm rate based on the feedback data. When the false alarm rate exceeds a preset threshold, the alarm decision parameters are adjusted to obtain an optimized alarm decision model.

[0080] The beneficial effects of this invention include: constructing a hierarchical multi-level verification funnel, which effectively filters noise caused by normal business fluctuations through verification of business perception baselines and related indicators; replacing the traditional continuous triggering mechanism with a multi-level verification funnel, thus accelerating fault response speed; distinguishing event scope through cross-customer verification and combining it with root cause localization in the business link, thus accelerating the fault investigation process; and possessing self-learning capabilities, which can continuously optimize alarm accuracy based on operation and maintenance feedback, achieving accurate, timely, and automated monitoring and alarms. Attached Figure Description

[0081] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0082] Figure 1 A flowchart illustrating the resilient monitoring and alarm method based on multi-level collaborative verification provided in this embodiment of the invention;

[0083] Figure 2 This is a diagram illustrating the overall architecture of a resilient monitoring and alarm system based on multi-level collaborative verification, as provided in an embodiment of the present invention.

[0084] Figure 3 A detailed flowchart of the first-level verification steps provided for embodiments of the present invention;

[0085] Figure 4 A detailed flowchart of the second-level verification steps provided in this embodiment of the invention;

[0086] Figure 5 A detailed flowchart of the third-level verification steps provided in this embodiment of the invention;

[0087] Figure 6 A detailed flowchart of the root cause localization steps provided in an embodiment of the present invention;

[0088] Figure 7 A flowchart illustrating the feedback self-optimization mechanism provided in an embodiment of the present invention;

[0089] Figure 8 This is a schematic diagram of the structure of an elastic monitoring and alarm system based on multi-level collaborative verification provided in an embodiment of the present invention. Detailed Implementation

[0090] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0091] Example 1

[0092] like Figure 1 and Figure 2 As shown, this invention provides a resilient monitoring and alarm method based on multi-level collaborative verification, comprising:

[0093] Step S1: Obtain the historical monitoring indicator data corresponding to the real-time monitoring indicator data, verify the historical monitoring indicator data, and obtain the verified historical monitoring indicator data.

[0094] Step S2: Calculate the baseline value based on the verified historical monitoring indicator data, calculate the deviation between the current monitoring indicator value in the real-time monitoring indicator data and the baseline value, determine whether the deviation is within the tolerance range, and obtain the first-level verification result;

[0095] Step S3: When the deviation indicated by the first-level verification result exceeds the tolerance range, obtain the associated indicator data related to the real-time monitoring indicator data, calculate the expected value of the real-time monitoring indicator data based on the associated indicator data, compare the current monitoring indicator value with the expected value, and obtain the second-level verification result.

[0096] Step S4: When the second-level verification result indicates that the change in the current monitoring indicator value cannot be explained by the change in the associated indicator data, obtain the same indicator data of the similar object group, compare the monitoring indicator distribution of the abnormal object with the monitoring indicator distribution of the similar object group, calculate the statistical test p value, and obtain the third-level verification result.

[0097] Step S5: When the p-value in the third-level verification result is less than the preset significance level threshold, obtain the upstream dependent service list based on the dependency relationship, query the health status of the upstream dependent service, and obtain the root cause localization result; when the p-value in the third-level verification result is greater than or equal to the preset significance level threshold, it is determined to be a group anomaly, confirmed as a global event alarm, and alarm information is directly generated.

[0098] Step S6: Generate alarm information based on the root cause localization result, push the alarm information, and receive feedback annotations on the alarm information to obtain feedback data;

[0099] Step S7: Calculate the false alarm rate based on the feedback data. When the false alarm rate exceeds a preset threshold, adjust the alarm decision parameters to obtain an optimized alarm decision model.

[0100] This invention provides a resilient monitoring and alarm method based on multi-level collaborative verification. With hierarchical verification and closed-loop optimization as its core, it aims to solve problems such as high false alarm rates, difficulty in root cause location, and delayed response in traditional monitoring and alarm systems, achieving more accurate alarms and automated operation and maintenance. The method first controls reliability at the data source: acquiring historical monitoring indicator data corresponding to real-time monitoring indicator data and verifying the historical data to provide high-quality data support for subsequent baseline calculations. Based on this, the first-level verification stage is entered, using the verified historical data to calculate the baseline value. By comparing the deviation between the current value and the baseline value of the real-time monitoring indicator, it is determined whether it is within the tolerance range, completing the first round of anomaly screening. If the deviation exceeds the tolerance range, the method further initiates the second-level verification, acquiring associated indicator data related to the real-time indicator, calculating the expected value of the real-time indicator based on the associated data, and comparing the current value with the expected value to eliminate "false anomalies" that can be explained by changes in associated indicators, thus improving the accuracy of anomaly detection.

[0101] Once the anomaly is confirmed to be genuine, the third-level verification involves obtaining data on similar metrics from a group of similar objects, comparing the metric distribution of the anomalous object with that of the group, and calculating a statistical p-value. If the p-value is less than a preset significance level threshold, it is determined to be an individual anomaly. Based on business dependencies, the health status of upstream dependent services is queried to locate the root cause of the failure. When the p-value is greater than or equal to the preset significance level threshold (usually set to 0.05 or 0.1), it indicates that the behavior of the current anomalous object has no significant statistical difference from the overall performance of its group. At this point, it is determined to be a group anomaly, confirming a global event affecting multiple customers or instances. These events are usually caused by platform-level failures, regional network problems, or planned maintenance activities. A global event alarm is immediately generated, containing information such as the affected scope, possible causes, and priority, and is directly pushed to the relevant operations and maintenance team through specific channels for rapid coordination and response, minimizing the overall impact on business. Finally, the method pushes alarm content containing root cause information or global anomaly information, while collecting feedback annotations from operation and maintenance personnel on the alarms, and calculating the false alarm rate based on the feedback data; when the false alarm rate exceeds the preset threshold, the alarm decision parameters are automatically adjusted and the alarm model is optimized, forming a complete closed loop of "data verification - multi-level verification - alarm push - feedback optimization", continuously improving alarm accuracy and fault handling efficiency, and reducing manual operation and maintenance costs.

[0102] Example 2

[0103] In this embodiment, the step of verifying the historical monitoring indicator data to obtain verified historical monitoring indicator data includes:

[0104] Acquire historical monitoring indicator data within a preset time window, wherein the historical monitoring indicator data is a sequence of historical monitoring indicators arranged in chronological order;

[0105] Calculate the mean and standard deviation of the historical monitoring indicator sequence data, and determine the 3-Sigma threshold range based on the mean and standard deviation;

[0106] Traverse each data point in the historical monitoring indicator sequence data, determine whether the value of each data point falls outside the 3-Sigma threshold range, and mark the data points that fall outside the 3-Sigma threshold range as historical anomalies;

[0107] The historical anomalies are supplemented using linear interpolation or spline interpolation methods to obtain verified historical monitoring index data.

[0108] Specifically, verifying historical monitoring indicator data is a crucial step in building an accurate baseline, ensuring that the data foundation for anomaly detection is not influenced by historical outliers. This process first acquires historical monitoring indicator data within a preset time window, typically choosing a range of 7-30 days to balance data sufficiency and timeliness. The time window selection must consider the characteristics of the business cycle; for example, for cyclical businesses, it should cover at least one complete cycle to ensure comprehensive capture of business patterns. This data is stored in time-series format, with each data point corresponding to a monitoring indicator value at a specific timestamp, and the sampling interval is typically 1-5 minutes.

[0109] The statistical characteristics of the historical monitoring indicator sequence data are then calculated, with the mean and standard deviation being the core. The mean is calculated by summing all data point values ​​and dividing by the total number of data points, reflecting the central trend of the indicator. The standard deviation is obtained by calculating the square root of the sum of squared deviations of each point from the mean, quantifying the degree of fluctuation of the indicator. These two statistics form the basis for constructing the 3-Sigma threshold range, defined as [mean - 3 × standard deviation, mean + 3 × standard deviation]. The 3-Sigma rule is based on the normal distribution theory, assuming that under normal conditions, approximately 99.7% of the data should fall within this range. This threshold setting is statistically conservative, ensuring that only significantly abnormal data points are marked.

[0110] After establishing the threshold range, each data point in the historical sequence is examined one by one. For a data value x(t) at time point t, if |x(t) - mean| > 3 × standard deviation, then the point is marked as a historical outlier. This judgment process is direct and efficient, and applicable to various types of indicator data. The marking process typically generates a Boolean array of labels, corresponding one-to-one with the original data points, for subsequent processing.

[0111] For marked historical outliers, instead of simple deletion, numerical replacement is performed to maintain data continuity. The replacement employs interpolation techniques, primarily linear interpolation and spline interpolation. Linear interpolation is suitable for indicators with gradual changes. For an outlier x(t), if its adjacent normal points are x(tk) and x(t+m), the replacement value is calculated as x(tk) + (x(t+m) - x(tk)) × k / (k+m), essentially assuming that the data within the interval containing the outlier changes linearly. Spline interpolation, on the other hand, is suitable for indicators with drastic fluctuations or significant nonlinear characteristics. By constructing a cubic spline function, a smoother transition is achieved, ensuring not only numerical continuity but also derivative continuity, generating a more natural replacement value.

[0112] The selection of interpolation methods considers multiple factors: the characteristics of index variation (linear or nonlinear), computational efficiency requirements (spline interpolation involves greater computational cost), and the distribution of outliers (concentrated or dispersed). For dense anomalies (such as multiple consecutive outliers), piecewise interpolation or interpolation assisted by other relevant indicators may be used to ensure the rationality of the replacement values.

[0113] After the above steps, the verified historical monitoring indicator data is obtained, which retains the time-series pattern and business characteristics of the original data while eliminating abnormal interference that could distort the baseline. This purified historical data forms a solid foundation for building an accurate and stable baseline model, directly improving the accuracy and reliability of subsequent anomaly detection. Data verification, as a preprocessing step, lays a good foundation for data quality for the entire multi-level verification and alarm system.

[0114] Example 3

[0115] like Figure 3 As shown, the process of calculating a baseline value based on the verified historical monitoring indicator data, calculating the deviation between the current monitoring indicator value in the real-time monitoring indicator data and the baseline value, determining whether the deviation is within the tolerance range, and obtaining the first-level verification result includes:

[0116] Based on the timestamp of the real-time monitoring indicator data, obtain the historical monitoring indicator data of the same period with the corresponding time granularity from the verified historical monitoring indicator data, and calculate the historical average and historical fluctuation range of the historical monitoring indicator data.

[0117] Obtain business context metadata, which includes marketing campaign intensity weight, seasonality index, and resource coefficient; and multiply the marketing campaign intensity weight, the seasonality index, and the resource coefficient to calculate the context coefficient;

[0118] The baseline value is obtained by multiplying the historical average value by the context coefficient.

[0119] Multiplying the historical fluctuation range by the context coefficient yields the dynamic tolerance range;

[0120] The deviation is obtained by calculating the absolute value of the difference between the current monitoring indicator value in the real-time monitoring indicator data and the baseline value;

[0121] Determine whether the deviation is less than or equal to the dynamic tolerance range. When the deviation is less than or equal to the dynamic tolerance range, it is determined to be a normal fluctuation. When the deviation is greater than the dynamic tolerance range, it is marked as a suspected anomaly and carries the baseline value, the deviation, and the context coefficient to obtain the first-level verification result.

[0122] In this embodiment, the implementation method of the first-level verification is the preliminary screening layer in the multi-level collaborative verification architecture. It aims to identify genuine anomalies that deviate from expectations, while filtering out false anomalies caused by normal business fluctuations. The innovation of the first-level verification is the introduction of business context awareness capabilities, enabling the calculation of dynamic baselines and adaptive tolerance.

[0123] The first level of verification begins with real-time monitoring metrics data. Historical monitoring metrics data of the same period, at the corresponding time granularity, are extracted from the verified historical monitoring metrics data. Time granularity refers to the time unit for data aggregation; different monitoring metrics require different time granularities. For example, for frequently changing system metrics (such as CPU utilization), a minute-level granularity is typically used; for medium-frequency changing application metrics (such as API response time), a 5-15 minute granularity may be used; and for low-frequency changing business metrics (such as daily active users), an hourly or daily granularity may be used. Choosing an appropriate time granularity is crucial for capturing the true fluctuation patterns of the metrics: too fine a granularity introduces excessive noise, while too coarse a granularity may mask important fluctuation signals.

[0124] Historical monitoring indicator data mainly comprises two key components: the historical average for the same period and the historical fluctuation range. The historical average for the same period is the average of historical data compared to the current point in time, taking into account the periodic characteristics of time. For example, for indicators with significant daily fluctuations, the average is calculated by selecting data from the same point in time corresponding to the current moment over the past n days (usually 7 or 14 days); for indicators with weekly fluctuations, the average is calculated by selecting data from the same day of the week and the same point in time over the past n weeks. This periodic comparison method can effectively filter out periodic fluctuations in the data, making anomaly detection more accurate.

[0125] Historical fluctuation range quantifies the amplitude of an indicator's fluctuation over the same historical period and forms the basis for constructing a tolerance range. Two methods are typically used to calculate historical fluctuation range: one is based on standard deviation, calculating the standard deviation of the data for the same historical period and multiplying it by a factor (usually 1.5-3) to obtain the fluctuation range; the other is based on quantiles, calculating the 10th and 90th quantiles of the data for the same historical period, and using the interval between these two as the fluctuation range. The first method assumes that the data is approximately normally distributed, is simple to calculate but sensitive to outliers; the second method does not rely on distribution assumptions, is more robust to outliers, but has higher computational complexity. The system will select the appropriate calculation method based on the characteristics of different indicators.

[0126] The innovation of this embodiment lies in the introduction of business context metadata, enabling the monitoring system to "sense" changes in the business environment and adjust expectations accordingly. Business context metadata mainly includes three core factors: marketing campaign intensity weight, seasonality index, and resource coefficient.

[0127] The marketing campaign intensity weight measures the expected impact of a current marketing campaign on system load. For example, promotional activities on e-commerce platforms significantly increase user traffic and transaction volume, thus affecting various system metrics; new version releases from software service providers increase the frequency of use of new features, etc. This weight is usually set in advance by the business department based on the scale of the campaign and the expected impact, and then input into the monitoring system through a configuration interface. The value range is usually [1.0, 5.0], where 1.0 represents no special campaign and 5.0 represents a very large-scale campaign (such as the "Double Eleven" e-commerce promotion).

[0128] Seasonal indices capture seasonal fluctuations in business, such as differences between weekdays and weekends, the beginning and end of the month, and holiday effects. These indices are typically calculated automatically through time-series analysis of historical data. Using time-series decomposition techniques, historical data is broken down into trend components, seasonal components, and residual components. A seasonal index lookup table is then built based on the seasonal components, assigning a seasonal coefficient to each specific point in time (such as a particular day of the week, month, or holiday). The seasonal index typically ranges from [0.8, 3.0], where a value less than 1 indicates a business slump (such as the off-season in traditional retail), and a value greater than 1 indicates a peak season.

[0129] The resource coefficient reflects the expected impact of changes in system resource allocation on performance metrics. For example, server expansion reduces single-machine load, thus affecting metrics such as CPU utilization and response time; the implementation of database read / write separation affects database query performance, etc. The resource coefficient is usually automatically calculated by the system based on resource change records, but can also be manually configured by operations and maintenance personnel. The value range is [0.5, 2.0], where less than 1 indicates an increase in resources (expected metric value decreases), and greater than 1 indicates a decrease in resources (expected metric value increases).

[0130] Multiply these three factors together to calculate the overall context coefficient. This multiplicative relationship reflects the combined effect of the factors: for example, if there is a large-scale marketing campaign (weight 3.0) and resource expansion (coefficient 0.7) at the same time, the final context coefficient is 3.0 × 0.7 = 2.1, indicating that although the expansion will partially offset the impact of the campaign, the overall system load is still expected to increase.

[0131] By multiplying the historical average response time by the context coefficient, the adjusted baseline value for the current business environment is calculated. For example, if the historical average response time for an API is 100ms and the current context coefficient is 1.5 (indicating increased business load), then the adjusted baseline value is 100ms × 1.5 = 150ms. This means that the expected response time in the current environment will be 50% higher than the historical average. This anticipated adjustment makes anomaly detection more accurate and avoids a large number of false anomalies during peak business periods.

[0132] Similarly, the historical fluctuation range is multiplied by the context coefficient to obtain the dynamic tolerance range. This step ensures that the tolerance range can also adaptively adjust as the business environment changes. For example, during peak business periods, the normal fluctuation range of the metric will be expanded accordingly; during off-peak business periods, the tolerance for fluctuations will be narrowed to capture smaller anomalies.

[0133] The deviation is calculated by taking the absolute value of the difference between the current monitoring indicator value and the baseline value in the verified monitoring indicator data. The deviation measures the degree to which the current value deviates from the expected baseline and is a direct basis for judging anomalies. Different deviation calculation methods may be used for different types of indicators: for indicators with clear absolute value meaning (such as response time), the absolute difference is usually used; for indicators with more meaningful relative value (such as error rate), the relative difference (i.e., the difference divided by the baseline value) may be used.

[0134] Finally, it is determined whether the deviation is within the dynamic tolerance range. If the deviation is less than or equal to the dynamic tolerance range, the current indicator value is considered to be within the normal business fluctuation range and will not trigger further verification; if the deviation exceeds the range, it is marked as a suspected anomaly, and key information such as the baseline value, deviation, and context coefficient are passed to the next verification layer to form a complete first-level verification result.

[0135] The innovation of Level 1 verification lies in its departure from the static thresholds or simple historical comparison methods used in traditional monitoring systems. It introduces the concept of business context, enabling it to understand and adapt to dynamic changes in business operations. This dynamic baseline and adaptive tolerance mechanism significantly improves the accuracy of anomaly detection, effectively filtering out a large number of "false anomalies" caused by normal business fluctuations, while maintaining sensitivity to genuine system anomalies. In practice, this mechanism reduces the false alarm rate in the initial screening stage, significantly alleviating the burden on operations and maintenance personnel and improving the availability and value of the monitoring system.

[0136] Example 4

[0137] like Figure 4 As shown, the step of obtaining associated indicator data related to the real-time monitoring indicator data, calculating the expected value of the real-time monitoring indicator data based on the associated indicator data, comparing the current monitoring indicator value with the expected value, and obtaining the second-level verification result includes:

[0138] Obtain a pre-configured association indicator mapping table, and determine the association indicator data that are associated with the real-time monitoring indicator data based on the association indicator mapping table;

[0139] Historical data of the related indicator data and historical data of the real-time monitoring indicator data are obtained. A linear regression algorithm is used to fit the relationship between the historical data of the related indicator data and the historical data of the real-time monitoring indicator data to obtain the regression coefficients and intercept parameters, and a regression model is constructed.

[0140] Obtain the current actual value of the associated indicator data, substitute the current actual value of the associated indicator data into the regression model, and calculate the expected value of the real-time monitoring indicator data.

[0141] Calculate the relative error between the current monitoring indicator value and the expected value, and determine whether the relative error is less than a preset error threshold.

[0142] When the relative error is less than the preset error threshold, it is determined that the change in the current monitoring indicator value can be explained by the change in the associated indicator data, and an alarm is filtered out. When the relative error is greater than or equal to the preset error threshold, it is determined to be a real anomaly and carries the expected value and the relative error to obtain the second-level verification result.

[0143] In this embodiment, the second-level verification is a crucial step in the multi-level collaborative verification architecture. It further filters out "pseudo-anomalies" that, although marked as suspected anomalies in the first-level verification, can actually be reasonably explained by changes in other related indicators through correlation analysis of associated indicators. This method is based on an important observation: in complex IT systems, many monitoring indicators have inherent correlations, and a change in one indicator often causes corresponding changes in other related indicators.

[0144] The second level of verification first requires obtaining a pre-configured correlation indicator mapping table. This mapping table is a structured knowledge base that defines the potential causal or correlational relationships between various monitoring indicators in the system. For example, increased web server request volume typically leads to higher CPU utilization, increased database query volume usually causes longer database response times, and increased network bandwidth usage may lead to increased data transmission latency. These mapping relationships can originate from three sources: first, pre-configuration by domain experts based on system architecture and business logic; second, statistical correlations automatically discovered from historical data through machine learning algorithms; and third, causal relationships automatically derived based on system call chains and dependencies.

[0145] In practice, the correlation metric mapping table is typically a multi-layered database or configuration file. It defines a set of potential correlated metrics for each monitoring metric and assigns a relevance weight to each correlated metric. For example, for the metric "API response time," correlated metrics might include "API request volume" (weight 0.8), "database query response time" (weight 0.7), and "CPU utilization" (weight 0.5). Based on this mapping table, the system identifies other metrics that are correlated with the current real-time monitoring metric data, forming a correlated metric dataset.

[0146] To quantify the relationship between the correlation indicator and the target indicator, historical data for both is needed. This historical data typically covers a period of 30 to 90 days, long enough to capture stable correlation patterns without being too long to make the model outdated. A linear regression algorithm is then used to fit the relationship between this historical data. Linear regression is a fundamental algorithm in statistics and machine learning; it assumes a linear relationship between the dependent variable (target indicator) and the independent variable (correlation indicator) and attempts to find the linear equation that best fits the observed data.

[0147] In univariate linear regression, the model takes the following form: ,in Represents target indicators. Represents related indicators, It is the regression coefficient (slope). It is the intercept. It is estimated using the least squares method. and The value of minimizes the sum of squared errors between the predicted and observed values. For cases with multiple correlated indicators, this is extended to multiple linear regression: ,in Representing different correlation indicators, These are the corresponding regression coefficients.

[0148] In practical applications, the potential multicollinearity problem among related indicators (i.e., correlations also exist between related indicators) is considered, and regularization techniques such as ridge regression or LASSO are used to address this issue, thereby improving the model's stability and generalization ability. Additionally, to capture nonlinear relationships, variations such as multinomial regression (introducing higher-order terms like x², x³, etc.) or piecewise linear regression may be employed.

[0149] Model quality is typically assessed using the coefficient of determination (R²), which represents the proportion of the dependent variable variance that the model can explain. R² ranges from 0 to 1, with values ​​closer to 1 indicating stronger explanatory power. In this system, a model is considered to have sufficient explanatory power and can be used for subsequent analysis only when the R² value exceeds a preset threshold (usually 0.6 or 0.7). Otherwise, the relationship between the correlation index and the target index is considered unstable, and the second-level validation is skipped, proceeding directly to the next level of validation.

[0150] After constructing the regression model, obtain the current actual values ​​of the related indicator data, substitute them into the regression model, and calculate the expected values ​​of the real-time monitoring indicator data. This expected value represents: what is the reasonable expected value of the target indicator under the current state of the related indicator? For example, if the current API request volume has increased by 30% compared to the same period in history, then according to the regression model, the API response time may be expected to increase by 15%.

[0151] Next, the relative error between the current monitoring indicator value and the expected value is calculated. The relative error is defined as: |(current value - expected value) / expected value|. The relative error measures the degree to which the current value deviates from the model's predicted value and is a core indicator for identifying anomalies. The use of relative error, rather than absolute error, allows the system to adapt to indicators of different magnitudes and provides a relatively fair assessment of large and small values.

[0152] The system determines whether the calculated relative error is less than a preset error threshold. This preset error threshold represents the maximum relative error the system can tolerate, typically set between 0.2 and 0.3 (i.e., 20%-30%). Setting this threshold requires balancing two types of errors: setting it too low may lead to many normal fluctuations being misjudged as abnormal, while setting it too high may result in missed reports of genuine anomalies. In practice, this threshold is dynamically adjusted based on the historical volatility of different indicators and their business importance.

[0153] If the relative error is less than a preset threshold, the system determines that the change in the current monitored indicator value can be reasonably explained by changes in related indicator data, and in this case, the system will filter out the alarm. For example, if the API response time increases by 20%, but the API request volume also increases by 40%, and according to the regression model, the expected response time should increase by about 22%, then the current change in response time (20%) is within the expected range and should not trigger an alarm. This filtering mechanism effectively reduces false alarms caused by normal load changes.

[0154] If the relative error is greater than or equal to a preset threshold, the change in the current monitored indicator is determined to be unexplainable by related indicators, thus confirming it as a genuine anomaly. The expected value and relative error information are then carried over to the second-level verification result and passed to the next verification layer. This additional information is highly valuable for subsequent root cause analysis and alarm handling.

[0155] The innovation of the second-level verification lies in its departure from simply treating "deviations from historical patterns" as anomalies. Instead, it attempts to find reasonable explanations through changes in related metrics. This approach significantly improves the accuracy of anomaly detection, especially in environments with dynamically changing business loads. For example, during e-commerce promotional events, many system metrics may fluctuate significantly. Traditional monitoring systems might trigger numerous false alarms, while this system can identify fluctuations that can be explained by factors such as increased traffic, thus effectively filtering out these "normal anomalies."

[0156] In practice, Level 2 verification can further reduce the false alarm rate while maintaining high sensitivity to unexplained real anomalies, significantly improving the practicality and reliability of the monitoring system. Furthermore, correlation analysis provides richer contextual information for anomalies, helping operations personnel to understand and locate the root cause of problems more quickly.

[0157] Example 5

[0158] like Figure 5 As shown, the process of obtaining the same indicator data of similar object groups, comparing the monitoring indicator distribution of abnormal objects with the monitoring indicator distribution of the similar object groups, calculating the statistical test p-value, and obtaining the third-level verification result includes:

[0159] Construct customer feature vectors that include industry type, business scale, functional preferences and geographical distribution, and normalize the customer feature vectors of all customers to obtain normalized customer feature vectors.

[0160] The locality-sensitive hashing algorithm is used to perform fast approximate clustering on the normalized customer feature vectors, and the clustering results are updated offline at preset time intervals to obtain customer clusters.

[0161] Assign a similar customer cluster ID to each customer in the customer cluster, and cache the list of member customers corresponding to the similar customer cluster ID;

[0162] Based on the real-time monitoring indicator data that are determined to be real anomalies in the second-level verification results, the corresponding anomaly object identifier is extracted, the similar customer cluster ID of the customer corresponding to the anomaly object identifier is determined, and other customers with the same similar customer cluster ID are obtained from the member customer list to obtain the similar object group.

[0163] Obtain the same indicator data of the similar object group within the same time period, and construct the monitoring indicator distribution of the similar object group;

[0164] Based on the real-time monitoring indicator data that are determined to be real anomalies in the second-level verification results, the monitoring indicator distribution of the anomaly object is constructed.

[0165] The Mann-Whitney U test method is used to compare the distribution of monitoring indicators of the abnormal object with the distribution of monitoring indicators of the similar object group, and the p-value is calculated.

[0166] Determine whether the p-value is less than a preset significance level threshold. When the p-value is greater than or equal to the preset significance level threshold, determine that the abnormal object and the group behavior are consistent and confirm it as a global event alarm. When the p-value is less than the preset significance level threshold, determine that it is an individual abnormality and carry the p-value and the monitoring index distribution of the similar object group to obtain the third-level verification result.

[0167] In this embodiment, the third-level verification is a high-level verification layer in a multi-level collaborative verification system. Its innovation lies in introducing a cross-customer collaborative analysis mechanism, which effectively distinguishes between individual anomalies and global events by comparing the behavioral differences between abnormal objects and groups of similar objects. This distinction is crucial for root cause analysis and problem solving: individual anomalies usually point to customer-specific configuration or usage issues, while global events may be widespread problems at the system platform or infrastructure level.

[0168] The first step in Level 3 verification is to construct a customer feature vector. A customer feature vector is a multi-dimensional data structure describing the characteristics of a customer's business and system, containing four main categories of features: industry type, business scale, functional preferences, and geographic distribution. Industry type is a categorical variable representing the customer's business sector, such as finance, e-commerce, manufacturing, healthcare, and education. This feature is typically encoded as a one-hot vector for easier subsequent mathematical processing. Business scale is a set of quantitative indicators reflecting the intensity and load level of the customer's system usage, including the number of users, average daily transaction volume, API call frequency, and data storage volume. Functional preferences describe the main functional modules and usage patterns of the system used by the customer; for example, a customer might heavily use data analysis functions while lightly using user management functions. Geographic distribution reflects the geographical coverage of the customer's business and the distribution of major users, which is particularly important for understanding network latency and regional issues.

[0169] Because these features have different dimensions and ranges, it is necessary to normalize the feature vectors of all customers, mapping each feature value to the same scale (usually the [0, 1] interval). For continuous variables (such as business scale indicators), the system typically uses Min-Max normalization: x normalized = (x - x min ) / (x max - x min ), or Z-score standardization: x standardized = (x - μ) / σ, where μ is the mean and σ is the standard deviation. For categorical variables (such as industry type), their one-hot encoding is usually preserved. Normalization yields a standardized customer feature vector, giving different features relatively equal weight in subsequent analyses.

[0170] To efficiently identify similar customer groups, the Locality-Sensitive Hashing (LSH) algorithm is used for fast approximate clustering. LSH is an algorithm optimized for similarity search in high-dimensional spaces. Its core idea is that similar vectors, after being mapped by a hash function, have a high probability of being mapped into the same "bucket". Compared with traditional clustering algorithms such as K-means, LSH has a significant efficiency advantage when processing large-scale high-dimensional data, and can complete approximate clustering in sublinear time, making it particularly suitable for dynamic environments that require frequent updates.

[0171] In the implementation, multiple families of hash functions are used, each consisting of multiple random projection vectors. The dot product of the customer feature vector and these projection vectors is binarized (1 for greater than zero, 0 for less than or equal to zero) to form a binary signature. Similar customer feature vectors will generate similar signatures and be grouped into the same bucket. By adjusting the number of hash function families and the number of functions in each family, query efficiency and clustering accuracy are balanced.

[0172] This clustering operation is typically updated offline at preset time intervals (such as weekly or monthly) to adapt to gradual changes in customer business characteristics while avoiding the system overhead of frequent recalculations. The clustering result is a series of customer clusters, each containing customers that are close to each other in the feature space.

[0173] Each customer cluster is assigned a unique cluster ID, and this ID is bound to each customer within the cluster. Simultaneously, a mapping table is maintained in memory or a distributed cache to record a list of all member customers corresponding to each cluster ID, enabling quick lookup of customers within the same cluster. This caching mechanism avoids redundant calculations of clustering results during anomaly detection, significantly improving system efficiency.

[0174] When the second-level verification determines that a monitoring metric is a genuine anomaly, the corresponding anomaly object identifier is extracted from this anomaly data. The anomaly object identifier is typically a customer ID or instance ID, uniquely identifying the specific object experiencing the anomaly. Based on this identifier, the cluster ID of the similar customer to which the anomaly object belongs is determined. Then, other customers with the same cluster ID are retrieved from the cached member customer list; these customers constitute the similar object group. The similar object group represents a group of customers with similar business characteristics and usage patterns to the anomaly object, and their behavioral patterns provide important reference for judging the prevalence of the anomaly.

[0175] Next, we obtain the same indicator data for similar target groups within the same time period. "Same indicator" refers to the same monitoring indicator as the one that caused the anomaly, and "same time period" usually refers to the most recent monitoring cycle (such as the past 5 minutes, 15 minutes, or 1 hour, depending on the indicator's collection frequency). Using this data, we construct the monitoring indicator distribution for similar target groups. This distribution reflects the normal behavioral patterns of customer groups with similar business characteristics regarding this indicator at the current point in time.

[0176] Simultaneously, based on the monitoring indicator data identified as genuine anomalies in the second-level verification, a monitoring indicator distribution for the anomalous objects is constructed. Since the data points for a single object may be limited, data from the anomalous objects over the most recent monitoring periods (such as data from the past hour) is typically collected to form a statistically significant distribution.

[0177] To compare the differences between the two distributions, the Mann-Whitney U test (also known as the Wilcoxon rank-sum test) is used. The Mann-Whitney U test is a non-parametric statistical test used to assess whether two independent samples come from the same distribution. Compared to parametric tests (such as the t-test), the Mann-Whitney U test does not assume that the data follows a normal distribution, is more robust to outliers and small samples, and is particularly suitable for scenarios such as monitoring data that may contain noise and outliers.

[0178] The Mann-Whitney U test compares the rank sums of two samples. The specific steps are as follows: First, merge the two samples and sort them by size; then, assign a rank to each data point; next, calculate the rank sum of both samples separately; finally, calculate the U statistic based on the rank sums, and obtain the corresponding p-value by looking up a table or approximating the result. The p-value represents the probability of observing the current or more extreme differences under the null hypothesis (that the two samples come from the same distribution). The smaller the p-value, the more significant the difference between the two distributions, and the more likely they are to come from different populations.

[0179] The system determines whether the calculated p-value is less than the preset significance level threshold. The significance level is a key parameter in statistical hypothesis testing, determining the strength of evidence required to reject the null hypothesis. In this system, the significance level is typically set to 0.05 or 0.01, corresponding to 95% or 99% confidence levels. Choosing this threshold requires balancing two types of errors: Type I error (incorrectly classifying samples from the same distribution as different) and Type II error (incorrectly classifying samples from different distributions as the same). In monitoring systems, to avoid missing important global events, a relatively conservative significance level (such as 0.05) is usually chosen.

[0180] If the p-value is greater than or equal to the preset significance level threshold, it indicates that the behavior of the abnormal object is not statistically significantly different from that of the similar object group. This means that the abnormal phenomenon is likely affecting not only a single customer but the entire similar customer group. In this case, the abnormality is confirmed as a global event alarm. This usually means that the problem may lie in the shared infrastructure, platform service, or local area network, requiring system-level response and handling.

[0181] If the p-value is less than the preset significance level threshold, it indicates that the behavior of the abnormal object differs significantly from that of the similar object group, meaning the abnormal phenomenon is likely only affecting that specific customer. In this case, the anomaly is classified as an individual anomaly, and information such as the p-value and the distribution of monitoring indicators for the similar object group is included to form the third-level validation result. This additional information is highly valuable for subsequent root cause analysis and problem localization.

[0182] The innovation of Level 3 verification lies in introducing a cross-customer collaborative analysis perspective. Instead of focusing solely on the historical behavior of individual customers, it compares it with the current behavior of similar customer groups. This approach effectively distinguishes between global events and individual anomalies, providing crucial information for problem classification and root cause localization. For example, when network infrastructure in a certain region experiences a problem, all customers in that region may simultaneously experience similar performance degradation. In this case, the system can identify it as a global event affecting the entire customer group, rather than a series of scattered individual anomalies, thus avoiding numerous duplicate alerts and guiding the operations team to find the root cause of the problem more quickly.

[0183] In practice, Level 3 verification significantly improves the accuracy of anomaly classification, reduces response delays caused by duplicate alarms and misclassification, and provides more targeted processing procedures for different types of anomalies, thus significantly enhancing the practicality and value of the monitoring system.

[0184] Example 6

[0185] like Figure 6 As shown, the process of obtaining the list of upstream dependent services based on dependency relationships, querying the health status of the upstream dependent services, and obtaining the root cause localization result includes:

[0186] Obtain business log data containing TraceID, analyze the business log data to mine business processes, extract service call relationships, and construct a business link dependency graph; wherein the business link dependency graph describes the call dependency relationship between services and applications.

[0187] Based on the real-time monitoring indicator data that are determined to be individual anomalies in the third-level verification results, the corresponding anomaly object identifier and anomaly service identifier are extracted.

[0188] Based on the business link dependency graph, query all upstream dependent services of the current abnormal service corresponding to the abnormal service identifier to obtain the list of upstream dependent services;

[0189] Parallel query of monitoring metric data of each upstream dependent service in the upstream dependent service list, determine whether each upstream dependent service in the upstream dependent service list has triggered an abnormal alarm, and obtain the health status of each upstream dependent service in the upstream dependent service list;

[0190] A rule-based reasoning engine is used to traverse the list of upstream dependent services. When the health status of any upstream dependent service is abnormal, the upstream dependent service is marked as the root cause service. When the health status of all upstream dependent services in the list is normal, the root cause is located as an internal or unmonitored third-party dependency of this service, and the root cause location result is obtained.

[0191] In this embodiment, root cause analysis is a core component of the intelligent monitoring and alarm system, solving the problem of "knowing what happened but not why" in traditional monitoring systems. When an anomaly is detected, it's not enough to simply tell "what happened"; it's also necessary to infer "why it happened." This embodiment details an automatic root cause analysis-based method that traces the upstream dependencies of the abnormal service to pinpoint the source of the problem, providing crucial guidance for rapid remediation.

[0192] The first step in root cause analysis is building a business dependency graph. A business dependency graph is a directed graph model that describes the call relationships between service components in a microservice architecture. It reflects the execution path of a business process and the dependencies between services. In modern distributed systems, a simple business request (such as a user placing an order) may require calls to dozens or even hundreds of microservices to complete, and an exception in any step can cause the entire business process to fail. A business dependency graph allows the system to "see" these complex call relationships, providing a foundation for anomaly tracing.

[0193] The core of building a business chain dependency graph is acquiring and analyzing business log data containing TraceIDs. A TraceID is a key concept in a distributed tracing system; it's a globally unique identifier generated when a business request enters the system and passed through all service calls during the entire request processing. When a business request sequentially passes through services A, B, and C, the logs generated by these services will all contain the same TraceID, enabling the system to correlate logs scattered across different services and reconstruct the complete call chain.

[0194] Business logs typically contain rich call information, such as call time, source service, target service, interface name, execution status, and execution time. By collecting and analyzing a large number of business logs containing TraceIDs, process mining algorithms are used to extract service call relationships. Process mining is a class of algorithms specifically designed to discover business processes from event logs, including alpha algorithms, heuristic mining algorithms, and frequent pattern-based mining algorithms. These algorithms can identify the sequence, frequency, and conditions of service calls from log data, thereby constructing a call path model that reflects the actual business flow.

[0195] In practice, distributed log collection tools (such as ELK Stack, Fluentd, etc.) are used to collect business logs from each service node, and combined with data from distributed tracing systems (such as Zipkin, Jaeger, etc.), the data is analyzed through batch processing or stream processing. Due to the massive amount of log data, the system typically adopts an incremental update strategy, processing new logs daily or weekly to gradually improve and update the dependency graph to adapt to the constantly evolving business processes.

[0196] A complete business dependency graph is a multi-level directed graph structure, where nodes represent services or applications and edges represent call relationships. Each edge also includes statistical information such as call frequency, average response time, and error rate, which provides important references for subsequent root cause inference. For example, if the error rate of service A calling service B has always been high, then when service A experiences an anomaly, the system will be more inclined to regard service B as a potential root cause.

[0197] Once an anomaly is detected and multi-level verification is completed, monitoring indicator data that is determined to be an individual anomaly is extracted from the third-level verification results. Furthermore, the corresponding anomaly object identifier and anomaly service identifier are extracted. The anomaly object identifier is typically a customer ID or instance ID, identifying the specific object where the anomaly occurred. The anomaly service identifier refers to the identifier of the specific service or component where the anomaly occurred, such as the service name or application ID.

[0198] Based on the previously constructed business dependency graph, query all upstream dependent services corresponding to the abnormal service identifier to obtain a list of upstream dependent services. Upstream dependent services refer to those services directly or indirectly called by the abnormal service; their performance or availability issues may propagate and affect the current service. For example, if the order service calls the inventory service and the payment service, then for the order service, the inventory service and the payment service are its upstream dependent services.

[0199] In distributed systems, failures often propagate downstream along the dependency chain, a characteristic known as the "avalanche effect": a failure in an upstream service can cause problems in downstream services that depend on it. Based on this characteristic, when a service is found to be abnormal, checking the health of its upstream dependent services is an effective way to find the root cause. If the upstream service is also abnormal, then the problem likely originates upstream rather than in the current service itself.

[0200] The system queries the monitoring metrics data of each service in the upstream dependency service list in parallel to determine whether these upstream services have also triggered abnormal alerts. Parallel querying is an optimization technique that allows queries to be sent to multiple monitoring data sources simultaneously, significantly reducing waiting time. In implementation, thread pools or asynchronous I / O techniques are typically used to manage these parallel query tasks, and reasonable timeout mechanisms are set to ensure that the query latency of a certain service does not block the entire root cause analysis process.

[0201] The query results retrieve the health status of each service in the upstream dependency service list. Health status is typically categorized into three levels: "Normal," "Warning," and "Abnormal," reflecting the overall operational status of the service. Health status assessment is based on multiple monitoring metrics, such as CPU utilization, memory usage, response time, and error rate, and may consider the weight and correlation of different metrics.

[0202] Once the health status of each upstream service is known, a rule-based reasoning engine is used to infer root causes. Rule-based reasoning is a classic method in artificial intelligence, using predefined IF-THEN rules to reason about known facts and draw conclusions. Compared with machine learning methods, rule-based reasoning has the advantages of clear logic and interpretable results, making it particularly suitable for fault diagnosis scenarios that require high transparency.

[0203] The inference engine traverses the list of upstream dependent services, applying predefined inference rules. The most basic rule is: when the health status of any upstream dependent service is abnormal, that upstream dependent service is marked as the root cause service. This rule is based on the principle of "fault propagation": if an upstream service is abnormal, it is likely the root cause of the current service's abnormality. When multiple abnormal upstream services exist, more complex rules are applied, such as considering factors like the severity of the abnormality, the duration of the abnormality, and historical reliability, or using topological sorting to determine the most upstream abnormal service in order to find the most likely initial point of failure.

[0204] If all upstream dependent services are found to be in normal health status after a complete review, the root cause will be identified as either "within this service" or "unmonitored third-party dependencies." "Within this service" indicates the problem likely stems from code defects, configuration errors, or insufficient resources within the service itself; "unmonitored third-party dependencies" refers to external components that the service depends on but are not under monitoring, such as third-party APIs or public cloud services. In this case, additional internal service metrics and log information may be added to help developers further investigate the issue.

[0205] It's important to note that in a microservice architecture, service dependencies can be very complex, even involving circular dependencies. To avoid getting stuck in infinite loops in such situations, inference engines typically set a maximum traversal depth limit and use access markers to avoid repeatedly analyzing the same service. They also consider the weight and criticality of dependencies, prioritizing the analysis of upstream dependencies that are highly coupled to the current service and called frequently.

[0206] The final output of root cause analysis is a structured root cause analysis result, containing the following key information: root cause service identifier (or marked "within this service"), root cause credibility score, dependency path from the root cause to the anomalous service, and relevant anomaly indicator data. This information not only indicates "where the problem lies," but also provides a chain of evidence for "why the problem is believed to lie there," providing clear direction for further investigation and remediation by operations and maintenance personnel.

[0207] Example 7

[0208] like Figure 7As shown, the step of generating alarm information based on the root cause localization result and pushing the alarm information includes:

[0209] Based on the root cause localization results, the root cause service identifier, the abnormal object identifier, and the abnormal service identifier are extracted.

[0210] Extract the names of abnormal indicators from the real-time monitoring indicator data;

[0211] Extract the deviation from the first-level verification results;

[0212] Extract the relative error from the second-level verification results;

[0213] Extract the p-value from the third-level verification result;

[0214] The root cause service name is determined based on the root cause service identifier. When the root cause location result indicates that the root cause is located within this service, the service name corresponding to the abnormal service identifier is used as the root cause service name. When the root cause location result indicates that the root cause is located in an upstream dependent service, the name of the upstream dependent service marked as the root cause service is used as the root cause service name.

[0215] The verification confidence level is calculated based on the deviation, the relative error, and the p-value; wherein the verification confidence level is a weighted comprehensive score of the three-level verification results.

[0216] Based on the root cause service name and the verification confidence level, processing suggestions are generated using preset processing suggestion generation rules; wherein the processing suggestions include suggested investigation directions and suggested processing measures;

[0217] Construct an alarm information data structure, which includes the alarm timestamp, the abnormal object identifier, the abnormal service name corresponding to the abnormal service identifier, the abnormal indicator name, the deviation, the root cause service name, the verification confidence level, and the processing suggestion;

[0218] The alarm information data structure is encapsulated in JSON format or message queue format and pushed to the operation and maintenance platform through API interface or message push service.

[0219] In this embodiment, the generation and push of alarm information is the final step of the intelligent monitoring system. It transforms the complex analysis results of the preceding steps into clear and useful information and promptly delivers it to relevant personnel. Unlike traditional monitoring systems that simply report "XX indicator exceeds the threshold," this embodiment details a method for constructing a rich and information-dense intelligent alarm mechanism, enabling maintenance personnel to quickly understand the essence of the problem and take targeted measures.

[0220] The first step in generating alarm information is to extract key information from the preceding analysis results. First, based on the root cause localization results, the root cause service identifier, the abnormal object identifier, and the abnormal service identifier are extracted. The root cause service identifier points to the source service most likely causing the current anomaly, the abnormal object identifier indicates the customer or instance experiencing the anomaly, and the abnormal service identifier specifies the service where the anomaly was directly observed. These three identifiers together constitute the "coordinate system" of the abnormal event, clarifying "who," "where," and "why" the problem occurred.

[0221] Next, extract the names of anomalous metrics from the real-time monitoring data. An anomalous metric name refers to the specific monitoring metric where the anomaly occurred, such as "CPU utilization," "memory usage," "response time," and "error rate." It clarifies "what" is the problem—that is, what specific anomaly was observed. Depending on the business importance and severity of the anomaly, multiple related anomalous metrics may be extracted simultaneously to provide a more comprehensive view of the problem.

[0222] To quantify the severity of anomalies, key numerical indicators are extracted from the validation results at each level: deviation is extracted from the first-level validation results, relative error from the second-level validation results, and p-value from the third-level validation results. These values ​​provide a quantitative assessment of the anomaly from different perspectives: deviation reflects the degree of deviation between the current value and the expected baseline; relative error indicates the difference between the current value and the predicted value based on correlation indicators; and the p-value quantifies the significance of the difference between the anomaly and similar group behaviors. These three indicators together constitute a multi-dimensional quantitative system for anomaly assessment, making the severity rating of alarms more objective and comprehensive.

[0223] Based on the root cause service identifier, the specific name of the root cause service is determined, making alarm information more user-friendly and easier to read. When the root cause localization result indicates that the problem originates within this service, the service name corresponding to the abnormal service identifier is used as the root cause service name; when the root cause is located in an upstream dependent service, the name of the upstream dependent service marked as the root cause is used. Service names are usually meaningful business terms, such as "order service," "payment gateway," and "user authentication service," which are easier for people to understand than technical identifiers.

[0224] To assess the credibility of the alert, a comprehensive validation confidence score is calculated based on the previously extracted deviation, relative error, and p-value. The validation confidence score is a weighted composite score of the three-level validation results, reflecting the degree of certainty that "this is indeed a real anomaly requiring attention." The calculation method typically uses a weighted average: the three indicators are normalized to the same scale (e.g., 0-100 points), and then weights are assigned according to the importance of each indicator to calculate the weighted average score. For example, deviation might be weighted at 30%, relative error at 40%, and p-value at 30% to obtain the final confidence score.

[0225] High-confidence alerts indicate that multiple verifications have confirmed the anomaly is real and worthy of attention, and operations personnel should prioritize its handling. Low-confidence alerts, on the other hand, indicate a degree of uncertainty and may require further manual judgment. Verifying confidence levels not only helps operations personnel prioritize and filter alerts but also serves as an important feedback signal for the system's self-learning and optimization.

[0226] Based on the root cause service name and verification confidence level, preliminary handling guidance is provided to operations and maintenance personnel through preset handling suggestion generation rules. The handling suggestions consist of two core parts: suggested investigation directions and suggested handling measures. The suggested investigation directions indicate the system components, configuration items, or log files that should be prioritized for inspection, such as "suggest checking database connection pool configuration" or "suggest analyzing the network connection status between service A and service B." The suggested handling measures are possible solutions proposed by the system based on historical experience, such as "consider restarting the service instance," "suggest scaling up service nodes," or "requires rolling back the most recent configuration changes."

[0227] The rule generation system for handling suggestions is a knowledge base that maps specific types of anomaly patterns to corresponding troubleshooting and handling strategies. This knowledge base can be derived from three sources: first, the encoding of expert experience, where domain experts transform handling experience into structured rules; second, the summarization of historical cases, where the system analyzes historical failures and their solutions to extract common patterns; and third, automatic discovery through machine learning, where algorithms automatically learn the correlation between anomaly patterns and effective handling measures by analyzing a large number of failure handling records.

[0228] In practice, rule generation for handling suggestions typically takes the form of decision trees or product rules. For example, for exceptions like "database connection timeout," if the root cause points to the database service itself and the verification confidence is high, the suggestion is to "check the database load and connection pool status, and consider scaling up the database instance"; if the root cause points to the application service, the suggestion is to "check the application's connection pool configuration and release mechanism, and consider optimizing database queries." This context-based intelligent suggestion greatly reduces the complexity of problem diagnosis, and is especially useful for less experienced operations and maintenance personnel.

[0229] With all the above information, a complete alarm information data structure is constructed. This data structure includes: alarm timestamp, abnormal object identifier, abnormal service name, abnormal indicator name, deviation, root cause service name, verification confidence level, and handling suggestions. The alarm timestamp records the exact time the abnormality was detected, which helps in time correlation analysis with other system events; the other fields contain the various information extracted and generated above, collectively forming a rich and clearly structured abnormality report.

[0230] This alarm information data structure will be encapsulated in a standardized data format for easy system integration and subsequent processing. The most commonly used format is JSON (JavaScript Object Notation), a lightweight data exchange format that is easy for humans to read and machines to parse, and is widely supported by various programming languages ​​and platforms. For scenarios requiring high throughput processing, the system may also use more compact message queue formats, such as Protocol Buffers, Avro, or Thrift, which offer higher serialization / deserialization efficiency and smaller data size.

[0231] Finally, alarm information is pushed through two main channels: API interfaces and push notification services. API interfaces are suitable for integration with other operations and maintenance platforms or ticketing systems, allowing these systems to actively invoke or passively receive alarm information and process it according to their own logic. Push notification services are responsible for delivering alarm information to relevant personnel in a timely manner. Common push channels include email, SMS, and enterprise instant messaging tools (such as Slack, WeChat Work, and DingTalk), selecting appropriate push channels and strategies based on the severity of the alarm, the time, and the responsible party's configuration.

[0232] In actual operation, alarm information push also considers a number of advanced features, such as alarm merging (merging similar alarms within a short period of time into one), alarm suppression (temporarily suppressing related alarms of downstream services when an alarm is detected in an upstream service), and shift rotation (sending alarms to the currently on-duty personnel according to a preset shift schedule). These features further reduce alarm noise, improve the accuracy of information delivery, and prevent maintenance personnel from being overwhelmed by a large number of duplicate or irrelevant alarms.

[0233] Example 8

[0234] The step of receiving feedback annotations for the alarm information and obtaining feedback data includes:

[0235] The operation and maintenance platform receives processing operations from operation and maintenance personnel on the alarm information data structure, including confirming it as a real anomaly or marking it as a false alarm.

[0236] The processing operation is stored as a feedback label in the feedback database, and the feedback data is obtained by associating it with the unique identifier of the alarm information data structure and the alarm rule identifier.

[0237] In this embodiment, the adaptive learning mechanism is a core feature of modern intelligent monitoring systems. It can continuously learn and optimize from operational practices, gradually improving the accuracy and usefulness of alarms. This embodiment elaborates on the feedback collection process, establishing a bridge from human expert knowledge to machine learning, and providing the necessary training data for the system's continuous self-optimization.

[0238] The starting point for feedback collection is receiving alarm information processing operations from operations and maintenance (O&M) personnel on the O&M platform. The O&M platform is the primary interface for O&M personnel to interact with the monitoring system, typically existing as a web application or specialized client software, providing functions such as alarm viewing, confirmation, processing, and analysis. When an alarm is pushed to the O&M platform, O&M personnel will handle it according to the actual situation. These processing operations are valuable expert feedback, reflecting the actual effectiveness of the alarm.

[0239] Handling operations mainly fall into two categories: confirming a genuine anomaly or marking it as a false alarm. Confirming a genuine anomaly indicates that the operations and maintenance personnel acknowledge it as a real problem requiring attention and action. This confirmation can take various forms, such as "confirm and handle," "assign to expert handling," or "create a work order for follow-up," all of which indicate that the alarm has captured a genuine system anomaly. Marking it as a false alarm indicates that the operations and maintenance personnel determine it is not a problem requiring real intervention; it may be due to unreasonable threshold settings, misinterpreting normal business fluctuations, or temporary environmental factors causing a false alarm. Besides the simple "mark as false alarm" operation, operations and maintenance personnel usually provide specific reasons for the false alarm, such as "normal business peak," "known maintenance activity," or "temporary testing."

[0240] In some systems, processing operations may also include finer-grained classifications, such as "partially confirmed" (an anomaly is confirmed but its severity is exaggerated), "known problem" (a real anomaly is confirmed but it is a known periodic problem), and "requires adjustment" (a real anomaly but the alarm rules need optimization). These refined classifications provide the system with richer learning signals, making optimization more targeted.

[0241] Regardless of the processing method chosen by the operations and maintenance personnel, these feedback tags will be stored in association with the original alarm information. Association storage means establishing a clear correspondence between feedback data and the alarm information that triggered that feedback, ensuring the system can accurately understand "which feedback corresponds to which alarm." This association is typically achieved through unique identifiers: each alarm information is assigned a globally unique ID, and the feedback data references this ID to establish the association.

[0242] The storage of feedback data typically includes three key elements: a unique identifier for the alarm information data structure, an alarm rule identifier, and feedback annotation content. The unique identifier for the alarm information data structure is the globally unique ID mentioned earlier, which uniquely identifies a specific alarm event. The alarm rule identifier identifies the specific rule that triggered the alarm; it is crucial for the system to understand "which types of rules are prone to false alarms." The feedback annotation content includes basic classification (real anomaly / false alarm), processing time, processing personnel, and possible processing notes.

[0243] This feedback data is stored in a dedicated feedback database, typically designed for high write frequencies and complex queries, such as a time-series or document database. The feedback database not only supports simple storage and retrieval but also complex aggregation analysis, such as calculating false alarm rates and confirmation rates by alarm rules, time periods, and handlers, providing data support for subsequent optimization analysis.

[0244] In practical implementation, feedback collection systems need to consider several practical issues. First, user experience optimization: feedback operations should be simple and intuitive, avoiding excessive mandatory input fields and reducing the operational burden on maintenance personnel. Second, the timeliness of feedback: maintenance personnel should be encouraged to provide feedback immediately after handling alarms, rather than accumulating a large number of unhandled alarms. Third, feedback consistency: different maintenance personnel may give different judgments for similar alarm situations, requiring standardized operating guidelines and regular consistency checks to reduce such discrepancies. Finally, feedback completeness: some alarms may be ignored or receive no clear feedback; mechanisms are needed to handle these "no feedback" situations, such as periodic reminders, automatic archiving, or inferences based on historical patterns.

[0245] The collection of feedback data not only provides a foundation for system optimization but also helps the operations and maintenance team manage knowledge and pass on experience. By analyzing historical feedback data, newly joined operations and maintenance personnel can quickly understand common problem types and their solutions, accelerating the learning curve; team managers can use feedback data analysis to identify system weaknesses and improvement opportunities, and carry out targeted technical or process optimizations.

[0246] Furthermore, feedback data provides a foundation for measuring operational efficiency. Key operational metrics can be calculated based on feedback data, such as average confirmation time (the average time from alarm triggering to confirmation by operations personnel), average resolution time (the average time from confirmation to being marked as resolved), and false alarm rate (the proportion of alarms marked as false alarms). These metrics help teams quantify operational efficiency, set improvement goals, and evaluate the effectiveness of optimization measures.

[0247] Overall, the feedback collection phase establishes a closed-loop learning system, enabling the monitoring system to continuously optimize through practice. It transforms the judgments of human experts into structured learning signals, laying a solid foundation for subsequent automated optimization. As feedback data accumulates and the "understanding" of the business and technical environment deepens, the accuracy and value of alarms also increase, ultimately achieving the goal of "getting smarter with use."

[0248] Example 9

[0249] The step of calculating the false alarm rate based on the feedback data, and adjusting the alarm decision parameters when the false alarm rate exceeds a preset threshold, yields an optimized alarm decision model, including:

[0250] Periodically scan the preset alarm rule set and extract the feedback label corresponding to each alarm rule in the alarm rule set within the preset time period from the feedback database;

[0251] Based on the feedback annotation, the total number of alarms triggered by each alarm rule in the alarm rule set within the preset time period and the number of alarms marked as false alarms are counted.

[0252] Based on the total number of alarms and the number of alarms marked as false alarms, calculate the false alarm rate for each alarm rule in the alarm rule set; wherein the false alarm rate is the ratio of the number of alarms marked as false alarms to the total number of alarms.

[0253] Determine whether the false alarm rate exceeds a preset false alarm rate threshold, filter out alarm rules whose false alarm rate exceeds the preset false alarm rate threshold, and obtain a set of alarm rules to be optimized;

[0254] For each alarm rule to be optimized in the set of alarm rules to be optimized, a parameter search space is defined based on each alarm rule to be optimized; wherein the parameter search space includes the historical window length, tolerance factor, and weight parameters of each verification level in the baseline calculation;

[0255] A Bayesian optimization algorithm is used to construct a Gaussian process surrogate model between parameter combinations and the false alarm rate based on the parameter search space. Candidate parameter combinations are then iteratively selected in the parameter search space using a data acquisition function.

[0256] The historical monitoring indicator data and corresponding feedback labels that have triggered alarms within the preset time period are obtained from the feedback database to obtain the verification historical dataset.

[0257] For each candidate parameter combination in the candidate parameter combinations, each candidate parameter combination is applied to each alarm rule to be optimized. The alarm triggering process is simulated based on the verification historical dataset, the simulated false alarm rate is calculated, and the Gaussian process proxy model is updated based on the simulated false alarm rate to obtain the updated Gaussian process proxy model.

[0258] Based on the updated Gaussian process proxy model, after a preset number of iterations, the candidate parameter combination with the lowest simulation false alarm rate is selected as the optimal parameter.

[0259] The optimal parameters are hot-updated to each alarm rule to be optimized in the online alarm decision engine through the configuration management system to obtain the optimized alarm decision model.

[0260] In this embodiment, the self-optimization mechanism is an advanced feature of the intelligent monitoring system, which can continuously adjust and optimize its decision model based on operational feedback, reducing manual intervention and improving alarm quality. This embodiment details an automatic parameter tuning method based on Bayesian optimization, which can systematically analyze feedback data, identify high false alarm rules, and automatically find the optimal parameter settings, enabling the continuous evolution of the monitoring system.

[0261] The first step in the self-optimization process is to periodically scan the preset alarm rule set and evaluate the actual performance of each rule. The alarm rule set is a collection of all currently active alarm rules in the monitoring system, and each rule defines specific anomaly detection logic and triggering conditions. Rule evaluation is typically performed at fixed time intervals (such as weekly or monthly). This interval needs to be long enough to accumulate a statistically significant sample size, but not so long that the problem persists without being optimized.

[0262] The evaluation process extracts feedback annotation data corresponding to each alarm rule within a preset time period (e.g., the past 30 days) from the feedback database. This data contains the handling results of each alarm by operations and maintenance personnel, clearly indicating which alarms were confirmed as real problems and which were marked as false alarms. The extraction process typically uses a specialized data query language (such as SQL or SQL-like languages) to perform precise queries based on alarm rule ID and time range, ensuring the integrity and accuracy of the data.

[0263] Based on the extracted feedback annotations, the total number of alarms triggered by each alarm rule within a preset time period and the number of alarms marked as false alarms are counted. The total number of alarms is the count of all alarm events triggered by the rule within the evaluation period, reflecting the triggering frequency of the rule; the number of alarms marked as false alarms is the count of alarms explicitly marked as "not requiring processing" by operations and maintenance personnel, reflecting the accuracy of the rule.

[0264] To quantify the accuracy of rules, the false alarm rate is calculated for each rule. The false alarm rate is the ratio of the number of alarms marked as false alarms to the total number of alarms, calculated as: False Alarm Rate = Number of False Alarms ÷ Total Number of Alarms. For example, if a rule triggered 100 alarms in the past 30 days, and 20 of them were marked as false alarms, then the false alarm rate for that rule is 20%. The false alarm rate is a key indicator for evaluating rule quality, directly reflecting the rule's usability and reliability.

[0265] After calculating the false positive rate, it is determined whether the false positive rate of each rule exceeds the preset false positive rate threshold. The preset false positive rate threshold is the maximum proportion of false positives that the system can tolerate, usually set between 10% and 30%. Setting this threshold requires balancing two aspects: a threshold that is too low may lead to too many rules being marked as needing optimization, increasing the system burden; a threshold that is too high may result in obviously problematic rules not being optimized in a timely manner. In practice, different types of rules may have different threshold settings. For example, rules for key business metrics may have a lower threshold (e.g., 10%), while rules for non-key metrics may use a more lenient threshold (e.g., 25%).

[0266] Alarm rules with false alarm rates exceeding a preset threshold are filtered out, forming a set of alarm rules to be optimized. This set contains all rules considered "poor quality" and requiring optimization, and is the object of subsequent optimization work. The size of the set of alarm rules to be optimized directly affects the workload and complexity of optimization. Typically, the system will set a priority ranking (e.g., based on false alarm rate from high to low), prioritizing the rules with the most prominent problems when resources are limited.

[0267] For each rule in the set of rules to be optimized, its parameter search space is first defined. The parameter search space is the set of all adjustable parameters and their value ranges, which defines the search boundary of the optimization algorithm. In monitoring systems, typical adjustable parameters include: the historical window length in baseline calculation (e.g., the past 7 days, 14 days, or 30 days), the tolerance factor (e.g., 1.5 times, 2 times, or 3 times the standard deviation), and the weight parameters of each validation level. Each parameter has its effective value range; for example, the historical window length may be between 1 and 60 days, and the tolerance factor may be between 1.0 and 5.0.

[0268] The definition of the parameter search space needs to comprehensively consider both theoretical rationality and practical constraints. Theoretically, the range of parameter values ​​should be broad enough to cover all possible optimal solutions; however, from a practical perspective, an excessively wide range of values ​​increases search difficulty and time cost, and some extreme parameter values ​​may lead to unacceptable side effects (such as an overly wide tolerance making the system insensitive to real anomalies). Therefore, an appropriate parameter range is usually set based on domain knowledge and historical experience to achieve a balance between coverage and efficiency.

[0269] After defining the search space, Bayesian optimization is used for parameter optimization. Bayesian optimization is an efficient black-box optimization method, particularly suitable for optimization problems with high computational costs, large parameter spaces, and unknown objective function shapes. Compared with traditional grid search or random search, Bayesian optimization utilizes information from previous evaluation points to guide subsequent searches, enabling faster convergence to the optimal solution and significantly reducing the number of evaluations and computational resource requirements.

[0270] The core of Bayesian optimization is constructing a Gaussian process surrogate model that connects parameter combinations to the objective metric (false alarm rate in this case). A Gaussian process is a probabilistic model that approximates an unknown function and provides an estimate of the uncertainty of the predicted value. In the optimization context, this model learns the mapping between parameters and the false alarm rate and is used to predict the possible performance of untried parameter combinations.

[0271] The two key components of a Gaussian process model are the mean function and the kernel function (also known as the covariance function). The mean function describes the prior expected response and is usually set to a constant; the kernel function describes the similarity or correlation between different points, and commonly used choices include the radial basis function (RBF) and the Matérn kernel. The choice of kernel function and its parameter settings have a significant impact on model performance and are usually selected based on problem characteristics and empirical knowledge.

[0272] After constructing the Gaussian process model, candidate parameter combinations are iteratively selected in the parameter search space using an acquisition function. The acquisition function is a key component of Bayesian optimization, balancing exploration and exploitation strategies: exploring unknown regions to obtain new information and using known information to find possible optimal solutions. Commonly used acquisition functions include Expected Improvement (EI), Upper Confidence Bound (UCB), and Probability of Improvement (PI). In this system, Expected Improvement is a common choice, comprehensively considering the mean and uncertainty of the predicted values, selecting the parameter combination most likely to bring improvement for the next evaluation.

[0273] To evaluate the actual performance of candidate parameter combinations, historical monitoring indicator data and corresponding feedback labels that have triggered alarms within a preset time period need to be obtained from the feedback database to form a validation historical dataset. The validation historical dataset is a collection containing historical monitoring data and its true labels (true anomalies / false alarms); it serves as the "ground truth" for evaluating the parameter combinations. This dataset should be large enough and representative, covering various types of normal fluctuations and true anomalies to ensure the generalization ability of the optimization results.

[0274] For each candidate parameter combination, it is applied to the alarm rule to be optimized. Based on the historical dataset for verification, the alarm triggering process is simulated, and the simulated false alarm rate is calculated. The simulated alarm triggering process refers to: rerunning the alarm detection logic using the candidate parameters, re-determining whether an alarm would be triggered at each time point in the historical data, and then comparing these "simulated alarms" with the actual feedback annotations to calculate the new false alarm rate. This simulation process is essentially "replaying" the alarm detection on historical data to evaluate how it would have performed if the new parameters had been used.

[0275] The calculation of the simulated false alarm rate considers multiple scenarios: correct triggering (simulated alarm triggering and actual genuine anomaly), false alarm (simulated alarm triggering but actual false alarm), missed alarm (simulated alarm not triggering but actual genuine anomaly), and correct non-triggering (simulated alarm not triggering and actual normal). While the false alarm rate is the primary optimization objective, the missed alarm rate is also considered to ensure that reducing false alarms does not significantly increase missed alarms. In some scenarios, a weighted composite metric (such as F1 score or a custom weighted sum) may be used as the optimization objective to balance the impact of false alarms and missed alarms.

[0276] The calculated simulated false alarm rate is used to update the Gaussian process surrogate model. Each evaluation of a new parameter combination yields a new observation point (parameter combination and corresponding false alarm rate), and this new information is used to update the Gaussian process model, continuously optimizing its understanding of the parameter space. Model updates typically involve re-estimating the kernel function parameters and updating the posterior distribution, which is a computationally intensive process. However, due to the relatively limited number of evaluations, the overall computational burden remains acceptable.

[0277] After updating the model, the next parameter combination to be evaluated is selected through a sampling function, and the next iteration begins. This iteration process continues for a preset number of times (usually between 20 and 100 times, depending on the size of the parameter space and computational resources), or until a specific stopping condition is met (such as continuous improvements below a threshold for multiple rounds). The number of iterations needs to be balanced between optimization quality and time cost: too few iterations may lead to suboptimal solutions, while too many iterations may result in diminishing returns and wasted computational resources.

[0278] After all iterations are completed, the optimal parameter is selected from all evaluated parameter combinations, assuming the lowest simulated false alarm rate. If the false alarm rates of multiple combinations are very close, other factors (such as false negative rate, parameter stability, etc.) may be considered in the final selection. The optimal parameter represents the best setting under the current data and optimization objectives, balancing false alarm control and anomaly detection sensitivity.

[0279] After obtaining the optimal parameters, they are hot-updated to the corresponding rules in the online alarm decision engine via the configuration management system. Hot updating refers to dynamically modifying the configuration while the system is running, without restarting the service or interrupting business operations. The configuration management system is an important component of an enterprise's IT infrastructure. It is responsible for the centralized management, version control, and distribution of various configurations, ensuring the security, consistency, and traceability of configuration changes. By updating parameters through the configuration management system, the system can ensure that changes are executed in an orderly manner, correctly recorded, and can be quickly rolled back when necessary.

[0280] After the parameters are updated hot, the optimized alarm decision model is officially put into use. The performance of these optimized rules will be continuously monitored, and new feedback data will be recorded to prepare for the next round of optimization. This cycle of continuous optimization enables the system to adapt to constantly changing business models and technological environments, maintaining high-quality alarm services.

[0281] In practical applications, self-optimizing systems also consider various advanced features and security measures. For example, a gradual deployment strategy: new parameters are first applied on a small scale to observe the effects, and then rolled out nationwide only after confirming there are no problems, reducing the risk of large-scale misconfiguration. Parameter history and comparative analysis: the history of parameter changes and corresponding performance changes are recorded to help understand the long-term effects and trends of parameter adjustments. Multi-objective optimization: multiple objectives such as false alarm rate, false negative rate, and alarm frequency are considered simultaneously to find parameter settings that balance various aspects. Manual review of abnormal optimization results: when the parameter changes suggested by the optimization algorithm are too large or the expected improvement is abnormally significant, a manual review process is triggered to avoid erroneous optimization caused by model anomalies.

[0282] Example 10

[0283] like Figure 8 As shown, the present invention also provides a resilient monitoring and alarm system based on multi-level collaborative verification, comprising:

[0284] The data acquisition and verification module 10 is used to acquire historical monitoring indicator data corresponding to real-time monitoring indicator data, verify the historical monitoring indicator data, and obtain verified historical monitoring indicator data.

[0285] The first-level verification module 20 is used to calculate the baseline value based on the verified historical monitoring indicator data, calculate the deviation between the current monitoring indicator value in the real-time monitoring indicator data and the baseline value, determine whether the deviation is within the tolerance range, and obtain the first-level verification result.

[0286] The second-level verification module 30 is used to obtain associated indicator data related to the real-time monitoring indicator data when the deviation indicated by the first-level verification result exceeds the tolerance range, calculate the expected value of the real-time monitoring indicator data based on the associated indicator data, compare the current monitoring indicator value with the expected value, and obtain the second-level verification result.

[0287] The third-level verification module 40 is used to obtain the same indicator data of similar object groups, compare the monitoring indicator distribution of abnormal objects with the monitoring indicator distribution of similar object groups, calculate the statistical test p value, and obtain the third-level verification result when the second-level verification result indicates that the change of the current monitoring indicator value cannot be explained by the change of the associated indicator data.

[0288] The root cause localization module 50 is used to obtain the upstream dependent service list based on the dependency relationship and query the health status of the upstream dependent services to obtain the root cause localization result when the p value in the third-level verification result is less than the preset significance level threshold; when the p value in the third-level verification result is greater than or equal to the preset significance level threshold, it is determined to be a group anomaly, confirmed as a global event alarm and directly generates alarm information.

[0289] The alarm push and feedback module 60 is used to generate alarm information based on the root cause localization result, push the alarm information, and receive feedback annotations on the alarm information to obtain feedback data.

[0290] The self-optimization module 70 is used to calculate the false alarm rate based on the feedback data. When the false alarm rate exceeds a preset threshold, the alarm decision parameters are adjusted to obtain an optimized alarm decision model.

[0291] In this system embodiment, the various functional modules jointly construct a complete multi-level collaborative verification elastic monitoring and alarm system. The system architecture adopts a modular design, with clearly defined responsibilities for each module. They collaborate to complete the entire process from data acquisition, multi-level verification, root cause localization to alarm push and self-optimization, achieving efficient and accurate monitoring and alarm functions.

[0292] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A resilient monitoring and alarm method based on multi-level collaborative verification, characterized in that, include: Obtain historical monitoring indicator data corresponding to real-time monitoring indicator data, verify the historical monitoring indicator data, and obtain verified historical monitoring indicator data. Based on the verified historical monitoring indicator data, a baseline value is calculated, and the deviation between the current monitoring indicator value in the real-time monitoring indicator data and the baseline value is calculated. It is then determined whether the deviation is within the tolerance range to obtain the first-level verification result. When the deviation indicated by the first-level verification result exceeds the tolerance range, the associated indicator data related to the real-time monitoring indicator data is obtained, the expected value of the real-time monitoring indicator data is calculated based on the associated indicator data, and the current monitoring indicator value is compared with the expected value to obtain the second-level verification result. When the second-level verification result indicates that the change in the current monitoring indicator value cannot be explained by the change in the associated indicator data, obtain the same indicator data of the similar object group, compare the monitoring indicator distribution of the abnormal object with the monitoring indicator distribution of the similar object group, calculate the statistical test p value, and obtain the third-level verification result. When the p-value in the third-level verification result is less than the preset significance level threshold, the upstream dependent service list is obtained based on the dependency relationship, the health status of the upstream dependent service is queried, and the root cause localization result is obtained; when the p-value in the third-level verification result is greater than or equal to the preset significance level threshold, it is determined to be a group anomaly, confirmed as a global event alarm, and alarm information is directly generated. Based on the root cause localization results, an alarm message is generated, the alarm message is pushed, and feedback annotations on the alarm message are received to obtain feedback data. The false alarm rate is calculated based on the feedback data. When the false alarm rate exceeds a preset threshold, the alarm decision parameters are adjusted to obtain an optimized alarm decision model.

2. The method according to claim 1, characterized in that, The step of verifying the historical monitoring indicator data to obtain verified historical monitoring indicator data includes: Acquire historical monitoring indicator data within a preset time window, wherein the historical monitoring indicator data is a sequence of historical monitoring indicators arranged in chronological order; Calculate the mean and standard deviation of the historical monitoring indicator sequence data, and determine the 3-Sigma threshold range based on the mean and standard deviation; Traverse each data point in the historical monitoring indicator sequence data, determine whether the value of each data point falls outside the 3-Sigma threshold range, and mark the data points that fall outside the 3-Sigma threshold range as historical anomalies; The historical anomalies are supplemented using linear interpolation or spline interpolation methods to obtain verified historical monitoring index data.

3. The method according to claim 2, characterized in that, The process involves calculating a baseline value based on the verified historical monitoring indicator data, calculating the deviation between the current monitoring indicator value in the real-time monitoring indicator data and the baseline value, determining whether the deviation is within the tolerance range, and obtaining the first-level verification result, including: Based on the timestamp of the real-time monitoring indicator data, obtain the historical monitoring indicator data of the same period with the corresponding time granularity from the verified historical monitoring indicator data, and calculate the historical average and historical fluctuation range of the historical monitoring indicator data. Obtain business context metadata, which includes marketing campaign intensity weight, seasonality index, and resource coefficient; and multiply the marketing campaign intensity weight, the seasonality index, and the resource coefficient to calculate the context coefficient; The baseline value is obtained by multiplying the historical average value by the context coefficient. Multiplying the historical fluctuation range by the context coefficient yields the dynamic tolerance range; The deviation is obtained by calculating the absolute value of the difference between the current monitoring indicator value in the real-time monitoring indicator data and the baseline value; Determine whether the deviation is less than or equal to the dynamic tolerance range. When the deviation is less than or equal to the dynamic tolerance range, it is determined to be a normal fluctuation. When the deviation is greater than the dynamic tolerance range, it is marked as a suspected anomaly and carries the baseline value, the deviation, and the context coefficient to obtain the first-level verification result.

4. The method according to claim 3, characterized in that, The process of acquiring associated indicator data related to the real-time monitoring indicator data, calculating the expected value of the real-time monitoring indicator data based on the associated indicator data, comparing the current monitoring indicator value with the expected value, and obtaining the second-level verification result includes: Obtain a pre-configured association indicator mapping table, and determine the association indicator data that are associated with the real-time monitoring indicator data based on the association indicator mapping table; Historical data of the related indicator data and historical data of the real-time monitoring indicator data are obtained. A linear regression algorithm is used to fit the relationship between the historical data of the related indicator data and the historical data of the real-time monitoring indicator data to obtain the regression coefficients and intercept parameters, and a regression model is constructed. Obtain the current actual value of the associated indicator data, substitute the current actual value of the associated indicator data into the regression model, and calculate the expected value of the real-time monitoring indicator data. Calculate the relative error between the current monitoring indicator value and the expected value, and determine whether the relative error is less than a preset error threshold. When the relative error is less than the preset error threshold, it is determined that the change in the current monitoring indicator value can be explained by the change in the associated indicator data, and an alarm is filtered out. When the relative error is greater than or equal to the preset error threshold, it is determined to be a real anomaly and carries the expected value and the relative error to obtain the second-level verification result.

5. The method according to claim 4, characterized in that, The process of obtaining the same indicator data of similar object groups, comparing the monitoring indicator distribution of abnormal objects with the monitoring indicator distribution of the similar object groups, calculating the statistical test p-value, and obtaining the third-level verification result includes: Construct customer feature vectors that include industry type, business scale, functional preferences and geographical distribution, and normalize the customer feature vectors of all customers to obtain normalized customer feature vectors. The locality-sensitive hashing algorithm is used to perform fast approximate clustering on the normalized customer feature vectors, and the clustering results are updated offline at preset time intervals to obtain customer clusters. Assign a similar customer cluster ID to each customer in the customer cluster, and cache the list of member customers corresponding to the similar customer cluster ID; Based on the real-time monitoring indicator data that are determined to be real anomalies in the second-level verification results, the corresponding anomaly object identifier is extracted, the similar customer cluster ID of the customer corresponding to the anomaly object identifier is determined, and other customers with the same similar customer cluster ID are obtained from the member customer list to obtain the similar object group. Obtain the same indicator data of the similar object group within the same time period, and construct the monitoring indicator distribution of the similar object group; Based on the real-time monitoring indicator data that are determined to be real anomalies in the second-level verification results, the monitoring indicator distribution of the anomaly object is constructed. The Mann-Whitney U test method is used to compare the distribution of monitoring indicators of the abnormal object with the distribution of monitoring indicators of the similar object group, and the p-value is calculated. Determine whether the p-value is less than a preset significance level threshold. When the p-value is greater than or equal to the preset significance level threshold, determine that the abnormal object and the group behavior are consistent and confirm it as a global event alarm. When the p-value is less than the preset significance level threshold, determine that it is an individual abnormality and carry the p-value and the monitoring index distribution of the similar object group to obtain the third-level verification result.

6. The method according to claim 5, characterized in that, The process of obtaining a list of upstream dependent services based on dependency relationships, querying the health status of the upstream dependent services, and obtaining root cause localization results includes: Obtain business log data containing TraceID, analyze the business log data to mine business processes, extract service call relationships, and construct a business link dependency graph; wherein the business link dependency graph describes the call dependency relationship between services and applications. Based on the real-time monitoring indicator data that are determined to be individual anomalies in the third-level verification results, the corresponding anomaly object identifier and anomaly service identifier are extracted. Based on the business link dependency graph, query all upstream dependent services of the current abnormal service corresponding to the abnormal service identifier to obtain the list of upstream dependent services; Parallel query of monitoring metric data of each upstream dependent service in the upstream dependent service list, determine whether each upstream dependent service in the upstream dependent service list has triggered an abnormal alarm, and obtain the health status of each upstream dependent service in the upstream dependent service list; A rule-based reasoning engine is used to traverse the list of upstream dependent services. When the health status of any upstream dependent service is abnormal, the upstream dependent service is marked as the root cause service. When the health status of all upstream dependent services in the list is normal, the root cause is located as an internal or unmonitored third-party dependency of this service, and the root cause location result is obtained.

7. The method according to claim 6, characterized in that, The process of generating and pushing alarm information based on the root cause localization results includes: Based on the root cause localization results, the root cause service identifier, the abnormal object identifier, and the abnormal service identifier are extracted. Extract the names of abnormal indicators from the real-time monitoring indicator data; Extract the deviation from the first-level verification results; Extract the relative error from the second-level verification results; Extract the p-value from the third-level verification result; The root cause service name is determined based on the root cause service identifier. When the root cause location result indicates that the root cause is located within this service, the service name corresponding to the abnormal service identifier is used as the root cause service name. When the root cause location result indicates that the root cause is located in an upstream dependent service, the name of the upstream dependent service marked as the root cause service is used as the root cause service name. The verification confidence level is calculated based on the deviation, the relative error, and the p-value; wherein the verification confidence level is a weighted comprehensive score of the three-level verification results. Based on the root cause service name and the verification confidence level, processing suggestions are generated using preset processing suggestion generation rules; wherein the processing suggestions include suggested investigation directions and suggested processing measures; Construct an alarm information data structure, which includes the alarm timestamp, the abnormal object identifier, the abnormal service name corresponding to the abnormal service identifier, the abnormal indicator name, the deviation, the root cause service name, the verification confidence level, and the processing suggestion; The alarm information data structure is encapsulated in JSON format or message queue format and pushed to the operation and maintenance platform through API interface or message push service.

8. The method according to claim 7, characterized in that, The step of receiving feedback annotations for the alarm information and obtaining feedback data includes: The operation and maintenance platform receives processing operations from operation and maintenance personnel on the alarm information data structure, including confirming it as a real anomaly or marking it as a false alarm. The processing operation is stored as a feedback label in the feedback database, and the feedback data is obtained by associating it with the unique identifier of the alarm information data structure and the alarm rule identifier.

9. The method according to claim 8, characterized in that, The step of calculating the false alarm rate based on the feedback data, and adjusting the alarm decision parameters when the false alarm rate exceeds a preset threshold, yields an optimized alarm decision model, including: Periodically scan the preset alarm rule set and extract the feedback label corresponding to each alarm rule in the alarm rule set within the preset time period from the feedback database; Based on the feedback annotation, the total number of alarms triggered by each alarm rule in the alarm rule set within the preset time period and the number of alarms marked as false alarms are counted. Based on the total number of alarms and the number of alarms marked as false alarms, calculate the false alarm rate for each alarm rule in the alarm rule set; wherein the false alarm rate is the ratio of the number of alarms marked as false alarms to the total number of alarms. Determine whether the false alarm rate exceeds a preset false alarm rate threshold, filter out alarm rules whose false alarm rate exceeds the preset false alarm rate threshold, and obtain a set of alarm rules to be optimized; For each alarm rule to be optimized in the set of alarm rules to be optimized, a parameter search space is defined based on each alarm rule to be optimized; wherein the parameter search space includes the historical window length, tolerance factor, and weight parameters of each verification level in the baseline calculation; A Bayesian optimization algorithm is used to construct a Gaussian process surrogate model between parameter combinations and the false alarm rate based on the parameter search space. Candidate parameter combinations are then iteratively selected in the parameter search space using a data acquisition function. The historical monitoring indicator data and corresponding feedback labels that have triggered alarms within the preset time period are obtained from the feedback database to obtain the verification historical dataset. For each candidate parameter combination in the candidate parameter combinations, each candidate parameter combination is applied to each alarm rule to be optimized. The alarm triggering process is simulated based on the verification historical dataset, the simulated false alarm rate is calculated, and the Gaussian process proxy model is updated based on the simulated false alarm rate to obtain the updated Gaussian process proxy model. Based on the updated Gaussian process proxy model, after a preset number of iterations, the candidate parameter combination with the lowest simulation false alarm rate is selected as the optimal parameter. The optimal parameters are hot-updated to each alarm rule to be optimized in the online alarm decision engine through the configuration management system to obtain the optimized alarm decision model.

10. A resilient monitoring and alarm system based on multi-level collaborative verification, characterized in that, include: The data acquisition and verification module is used to acquire historical monitoring indicator data corresponding to real-time monitoring indicator data, verify the historical monitoring indicator data, and obtain verified historical monitoring indicator data. The first-level verification module is used to calculate the baseline value based on the verified historical monitoring indicator data, calculate the deviation between the current monitoring indicator value in the real-time monitoring indicator data and the baseline value, determine whether the deviation is within the tolerance range, and obtain the first-level verification result. The second-level verification module is used to obtain associated indicator data related to the real-time monitoring indicator data when the deviation indicated by the first-level verification result exceeds the tolerance range, calculate the expected value of the real-time monitoring indicator data based on the associated indicator data, compare the current monitoring indicator value with the expected value, and obtain the second-level verification result. The third-level verification module is used to obtain the same indicator data of similar object groups when the second-level verification result indicates that the change of the current monitoring indicator value cannot be explained by the change of the associated indicator data, compare the monitoring indicator distribution of the abnormal object with the monitoring indicator distribution of the similar object group, calculate the statistical test p value, and obtain the third-level verification result. The root cause localization module is used to obtain the upstream dependent service list based on the dependency relationship and query the health status of the upstream dependent services to obtain the root cause localization result when the p value in the third-level verification result is less than the preset significance level threshold; when the p value in the third-level verification result is greater than or equal to the preset significance level threshold, it is determined to be a group anomaly, confirmed as a global event alarm and directly generates alarm information. The alarm push and feedback module is used to generate alarm information based on the root cause localization results, push the alarm information, and receive feedback annotations on the alarm information to obtain feedback data. The self-optimization module is used to calculate the false alarm rate based on the feedback data. When the false alarm rate exceeds a preset threshold, the alarm decision parameters are adjusted to obtain an optimized alarm decision model.

Images