An abnormality alarm method based on an improved abnormality detection algorithm

Abstract

The present disclosure belongs to the field of detection algorithm, and particularly relates to an abnormality alarm method based on an improved abnormality detection algorithm, comprising: performing data collection on an index source to obtain real-time index data, and storing historical index data and real-time index data to be calculated threshold; according to a threshold A set by an artificial, weighting the historical index data within the threshold A, performing abnormality detection on the historical index data based on the weighted abnormality detection algorithm, calculating a threshold B, and outputting a data abnormality label according to the threshold B; judging the index type of the historical index data, obtaining an alarm threshold range according to the index type, and storing the alarm threshold range; extracting the real-time index data and the alarm threshold range, and triggering alarm information when any kind of index data is within the alarm threshold range. The present application realizes automatic calculation and update of index threshold, and avoids the problem that the standard deviation is large when the index historical data fluctuates greatly, and the calculated threshold exceeds the reasonable interval.

Description

Technical Field

[0001] The present invention belongs to the field of detection algorithms, and in particular relates to an abnormality alarm method based on an improved abnormality detection algorithm. Background Art

[0002] Anomaly detection is the process of identifying data points within a dataset or system that fall outside of normal ranges. During data analysis or through machine learning algorithms, anomaly detection flags instances within the bulk of the data that don't conform to common patterns or statistical models. Anomalies can manifest as outliers, unexpected changes, or errors, depending on the type of data being analyzed and any predefined parameters set. Anomaly detection is valuable because it can quickly and effectively identify potential issues or threats, maintaining system integrity and reliability.

[0003] Currently, indicator thresholds are typically determined manually based on experience. These indicators generally refer to metrics used to monitor service status and quality within information services, such as service availability and latency. When service quality issues arise, corresponding indicators will decline, triggering threshold alerts and prompting personnel to perform maintenance to maintain high system quality. The typical process involves first specifying a standard threshold based on service requirements. During system operation, each threshold is then adjusted based on actual conditions, ultimately resulting in a comprehensive indicator threshold system.

[0004] When the number of indicators is small or the indicators do not change much, it is more feasible to manually specify the thresholds. However, when the number of indicators that need to be monitored is large, for example, there are thousands of indicators that need to be monitored and updated regularly, it is difficult for humans to efficiently confirm the thresholds.

[0005] Chinese invention patent specification CN202211665482 discloses a method for confirming alarm thresholds through historical data, the main technical feature of which is: based on the historical sampling data of the indicator, the statistical value standard deviation is calculated as the basis for threshold calculation. This method confirms the initial alarm threshold range based on the data obtained by sampling the historical indicator data in the standard time dimension and the specific time dimension, and then adjusts the initial alarm threshold range based on the data confirmed by the special scenario alarm, so as to obtain an alarm threshold range that meets the special scenario and exceeds the reasonable range, thereby reducing false alarms and achieving the purpose of accurate alarms. However, when the historical indicators fluctuate greatly, the standard deviation obtained is large. At this time, the threshold calculated based on the baseline value and the standard deviation will seriously exceed the normal situation, so it is not suitable for scenarios with large indicator fluctuations. Summary of the Invention

[0006] In view of the above problems in the prior art, when the number of monitored indicators is large, the workload of threshold setting according to manual experience is large, and the updating speed is slow; when the standard deviation of statistical indicators is used to determine the threshold, the standard deviation is large when the historical data fluctuation is large, and the calculated threshold exceeds the reasonable interval, the problem to be solved by the present disclosure is to provide an abnormal alarm method based on an improved abnormal detection algorithm, comprising:

[0007] Data acquisition is performed on the indicator source to obtain real-time indicator data, and historical indicator data and real-time indicator data to be calculated are stored; the historical indicator data is a set of N kinds of indicator data collected from the same indicator source;

[0008] According to the threshold A set by the manual, the historical indicator data within the threshold A is weighted, and the historical indicator data outside the threshold A is kept by default; the threshold A includes N thresholds, corresponding to the N kinds of indicator data in the historical indicator data;

[0009] The weighted abnormal detection algorithm is used to detect the abnormality of the historical indicator data, and a threshold B is calculated, and a data abnormality label is output according to the threshold B, and a normal value data of the historical indicator data is obtained;

[0010] Determine the indicator type of the historical indicator data, and according to the indicator type, obtain an alarm threshold range based on the data abnormality label and store the alarm threshold range;

[0011] Extract the real-time indicator data and the alarm threshold range, and when any kind of indicator data is within the alarm threshold range, trigger an alarm information.

[0012] The weighted abnormal detection algorithm comprises: weighted data = original data x weight.

[0013] The indicator type includes: interval type indicator, positive type indicator, and negative type indicator.

[0014] The alarm threshold range is determined according to the indicator type, comprising:

[0015] When the indicator type is an interval type indicator, the maximum value of the normal value data is taken as the upper limit of the threshold, the minimum value is taken as the lower limit of the threshold, and the alarm threshold range is taken as the range above the upper limit of the threshold and below the lower limit of the threshold;

[0016] When the indicator type is a positive type indicator, the minimum value of the normal value data is taken as the lower limit of the threshold, and the alarm threshold range is taken as the range below the lower limit of the threshold;

[0017] When the indicator type is a negative type indicator, the maximum value of the normal value data is taken as the upper limit of the threshold, and the alarm threshold range is taken as the range above the upper limit of the threshold;

[0018] The historical index data has the same value in any one of the historical time, and whether it is a normal value is judged according to the threshold A set by artificial, if all are out of the threshold A, it is not used, and the alarm information is triggered; if all are in the threshold A, the alarm threshold range of the index is determined according to the index type:

[0019] When the index type is an interval index, the alarm threshold range is taken from the range outside the historical index data which is the same in the historical time;

[0020] When the index type is a positive index, the alarm threshold range is taken from the range below the historical index data which is the same in the historical time;

[0021] When the index type is a negative index, the alarm threshold range is taken from the range above the historical index data which is the same in the historical time.

[0022] In the second aspect, the application further provides a system for the abnormal alarm method, characterized by comprising:

[0023] The monitoring index collection module is used for collecting the monitoring index of the monitoring object according to the preset collection frequency, collection object and the like, and transmitting the monitoring index to the data storage module for subsequent analysis.

[0024] The data storage module is used for bearing the data storage work, and stores the index data from the collection module and the alarm threshold range after calculation and analysis, for the monitoring alarm module.

[0025] The task scheduling module is used for automatically scheduling the task execution according to the pre-set task execution information, including extracting the historical data from the data storage, calculating the threshold, storing the threshold to the storage module and the like.

[0026] The monitoring alarm module is used for regularly extracting the alarm threshold range from the data storage server as the alarm information trigger threshold, and regularly extracting the real-time index data from the data storage module and comparing the real-time index data with the alarm threshold, if the real-time index data is in the alarm threshold range, the alarm is triggered.

[0027] The threshold calculation module is used for processing the index historical data according to the threshold calculation algorithm and the index historical data, calculating the threshold corresponding to the index, obtaining the alarm threshold range combined with the index type, and finally storing the alarm threshold range to the output storage module.

[0028] Compared with the prior art, the application can obtain the following technical effects:

[0029] 1. The application realizes the automation of index threshold calculation and update, can be applied to the automation calculation scene of large-scale monitoring of various types of indexes, and introduces artificial prior experience by improving the weight part of the abnormal detection algorithm, so that the abnormal label interval output by the algorithm is closer to the range of normal values, avoiding the output index threshold from deviating from the reasonable range in the case of more abnormal values.

[0030] 2. In the application, the index historical data is used for abnormal detection when obtaining the index alarm threshold, the normal value interval in the index is identified according to the abnormal detection result, and the index threshold is set according to the index normal value interval. Avoid the problem that the standard deviation is large when the index historical data fluctuates greatly, and then the calculated threshold exceeds the reasonable interval. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The figure is a framework diagram of an abnormal alarm system based on a preferred embodiment of the application.

[0032] Figure 2 The figure is a flow chart of an abnormal alarm method based on an improved abnormal detection algorithm based on a preferred embodiment of the application.

[0033] Figure 3 The figure is a flow chart of an abnormal alarm method based on an improved abnormal detection algorithm based on a preferred embodiment of the application. DETAILED DESCRIPTION

[0034] The best mode of the application will be described below by examples, and it should be understood that the specific embodiments herein are used to explain the application in detail, and should not be understood as a limitation of the application. It should be noted that various changes and modifications can be made within the scope of the principles and cores of the application, and these changes should be considered as belonging to the protection scope of the application. The specific implementation steps of the application are described in detail in combination with the drawings.

[0035] The embodiments of the specification provide an abnormal detection method and device, and first, the system architecture of the embodiments of the specification is described.

[0036] Figure 1 The figure is a system overall framework of an abnormal alarm method based on an improved abnormal detection algorithm based on the application. As shown in the figure, the system can include: Figure 1

[0037] Monitoring index acquisition module:

[0038] The monitoring index acquisition module automatically acquires the monitoring index of the monitoring object according to the preset acquisition frequency, acquisition object, etc., and transmits it to the data storage module for subsequent analysis.

[0039] ​The monitoring object is generally the service state and quality in information service, and the monitoring index is the corresponding service availability rate and delay rate data. The collection frequency should not be too low in order to calculate a suitable threshold in abnormality detection.

[0040] The data storage module:

[0041] The data storage module undertakes data storage work, and stores the index data from the collection module on one hand, and stores the alarm threshold range after calculation and analysis on the other hand for the monitoring and alarm module.

[0042] The task scheduling module:

[0043] The task scheduling module automatically schedules task execution according to the pre-set task execution information, including extracting historical data from the data storage, calculating the threshold, and storing the threshold to the storage module.

[0044] The monitoring and alarm module:

[0045] The monitoring and alarm module regularly extracts the alarm threshold range from the data storage server as the alarm information triggering threshold, and regularly extracts the real-time index data from the data storage module and compares it with the alarm threshold. If it is within the alarm threshold range, an alarm is triggered.

[0046] The threshold calculation module:

[0047] The threshold calculation module processes the index historical data according to the threshold calculation algorithm and the index historical data, calculates the threshold corresponding to the index, obtains the alarm threshold range in combination with the index type, and finally stores the alarm threshold range to the output storage module.

[0048] The specific implementation of the abnormality alarm method of the present application will be described below in combination with the abnormality detection process. Figure 2 The abnormality alarm method based on the improved abnormality detection algorithm includes the following steps:

[0049] Step S1: data collection work is performed on the index source to obtain real-time index data, and historical index data and real-time index data to be calculated threshold are stored; the historical index data is a set of N kinds of index data collected from the same index source;

[0050] Step S11: the task scheduling module issues a collection command to notify the collection module.

[0051] Step S12: the collection module receives the command, performs data collection work on the index source, and simultaneously stores it in the database.

[0052] Specifically, the collection module automatically collects the monitoring index of the monitoring object according to the pre-set collection frequency, collection object, etc., and transmits it to the data storage module for subsequent analysis.

[0053] The collected monitoring index data is generally corresponding to the service state, quality, etc. in the information service, including but not limited to delay rate, compliance rate, completeness rate, etc. historical index data and real-time index data that need to be detected.

[0054] Step S2: According to the threshold A corresponding to N kinds of indexes set by artificial, the historical index data within the threshold A is weighted, and the historical index data outside the threshold A is kept as default.

[0055] In a specific embodiment, the specific steps of S2 are:

[0056] Step S21: The task scheduling module notifies the threshold calculation module to perform threshold calculation work according to the timing task.

[0057] Step S22: Obtain the historical index source data to be calculated threshold.

[0058] After the threshold calculation module receives the corresponding notification, it initiates the data extraction work and obtains the basic data.

[0059] The historical index source data of the threshold to be calculated for anomaly detection in the storage module generally has at least one period of historical data.

[0060] The collected data can include delay rate, compliance rate, completeness rate, etc. historical index data that need to be detected.

[0061] Step S23: Preprocess the source data to obtain historical index data.

[0062] The threshold calculation module preprocesses the source data to remove irrelevant data such as empty data, text data, etc. to form a data set.

[0063] Step S24: Determine whether the historical index data is the same value, if the historical index data is the same value within any kind of index historical time, according to the threshold A set by artificial, determine whether the historical data of the index is a normal value, if all outside the threshold A, do not use, and notify the alarm module to trigger the alarm information;

[0064] If all are within the threshold A, execute the following steps:

[0065] Step S25: Determine the index type of the historical index data.

[0066] Among them, the index type includes positive index, negative index and interval index.

[0067] Among them, the positive index is the higher the index value, the better the performance of the index, the negative index is the lower the index value, the better the performance of the index, and the interval index is the value within a certain fixed interval is considered as the best data.

[0068] For example, the compliance rate is a positive type index, the delay rate is a negative type index, and a certain parameter that is not too large or too small is an interval type index.

[0069] Step S26: If the historical time of the historical index data is the same value and is within the threshold value A, the alarm threshold range of the index is determined according to the index type, and the threshold calculation module stores the alarm threshold range into the monitoring alarm module:

[0070] When the index type is an interval type index, the alarm threshold range is taken from the range outside the historical index data;

[0071] When the index type is a positive type index, the alarm threshold range is taken from the range below the historical index data;

[0072] When the index type is a negative type index, the alarm threshold range is taken from the range above the historical index data.

[0073] Step S3: Abnormality detection is performed on the historical index data based on a weighted abnormality detection algorithm, a threshold value B is calculated, a data abnormality label is output according to the threshold value B, and normal value data of the historical index data is obtained;

[0074] In a specific embodiment, if the historical time of the historical index data has different values, the threshold calculation module performs step S3, and step S3 includes:

[0075] Step S31: The default weight of the historical data is 1, the data within the threshold value A is weighted according to the manually set threshold value A, the weighted data = original data x weight, and the data outside the threshold value A remains default.

[0076] Step S32: Abnormality detection is performed on the historical index data according to the abnormality detection algorithm, and a data abnormality label is output;

[0077] Specifically, the abnormality detection algorithm can be a commonly used abnormality detection algorithm such as a clustering-based method or a statistical-based method.

[0078] In an embodiment of the present application, the abnormality detection algorithm used is an isolation forest algorithm, which is specifically as follows:

[0079] A certain number of samples are randomly extracted from the entire historical data set to form a subsample set, which is prepared for tree construction.

[0080] It should be noted that the size of the subsample set is usually much smaller than the size of the original historical data set, which can limit the size of the tree and reduce the computational complexity.

[0081] A tree is constructed for each subsample set, that is:

[0082] For the historical index data in the subsample set, at each node, the algorithm randomly selects a split value, and samples are divided into the left or right subtree according to the split value. This operation is repeated in the subtree until the tree reaches a defined height, the number of samples in the node reaches a certain number, or the values of all samples are the same.

[0083] For example, for the historical index data in the subsample set, at each node, the algorithm randomly selects a split value, and historical index data less than the split value is divided into the left subtree, and historical index data greater than the split value is divided into the right subtree. In the left and right subtrees, the split value is continued, historical index data less than the split value is divided into the left subtree of the subtree, historical index data greater than the split value is divided into the right subtree of the subtree, and the operation is repeated until the tree reaches a defined height, the number of samples in the node reaches a certain number, or the values of all samples are the same.

[0084] It should be noted that the height of the tree is limited because points with shorter path lengths are more likely to be outliers, and normal points with long path lengths are not the focus of anomaly detection.

[0085] Since the cutting process of each tree is completely random, several trees need to be established, and then the average value of the split results of each tree is calculated to make the results converge. After the specified number of isolated trees is constructed according to the above steps, the isolated forest is formed.

[0086] The path length of the index data in each isolated tree is calculated, and the average path length is calculated to obtain the anomaly score of each data point.

[0087] The anomaly score s is calculated by the following formula:

[0088]

[0089] Where h(x) is the height of an index data in a tree, that is, how many edges need to be passed to reach the leaf node from the root node of the tree.

[0090] E(h(x)) is the average height of the index data x in all trees, and the lower the height, the higher the anomaly score.

[0091] c(n) is the average value of the path length when the number of samples in the subsample set is n, which is used to normalize the path length h(x) of the index data x.

[0092] If E(h(x))→0, then s→1, which means that the average height of the x data point in all trees tends to 0. Since the data that is isolated first, that is, the data close to the root node, is more likely to be an outlier, it can be considered as an outlier.

[0093] Similarly, when E(h(x))→n-1, s→0, which means that the average height of the data points in all trees tends to n-1. The data points tend to be at the bottom of the tree, i.e., the maximum depth, and are not easily isolated and divided out, and can be considered as normal points.

[0094] According to the calculated abnormal score, an abnormal score threshold B is set to determine which data points are abnormal.

[0095] For abnormal data, output a data abnormality label.

[0096] Step S4: judging the index type of the historical index data, obtaining an alarm threshold range based on the data abnormality label according to the index type, and storing the alarm threshold range;

[0097] In specific embodiments, the specific steps of S4 include:

[0098] Step S41: judging the index type of the historical index data;

[0099] The index type includes a positive type index, a negative type index, and an interval type index.

[0100] The positive type index is an index whose higher index value indicates better performance, the negative type index is an index whose lower index value indicates better performance, and the interval type index is an index whose value within a certain fixed interval is considered to be the best data.

[0101] Step S42: calculating a threshold according to the index type and whether the index historical data is the same within a historical time, and outputting an alarm threshold range.

[0102] If the index historical data is different within a historical time, the alarm threshold range is determined according to the index type:

[0103] When the index type is an interval type index, the maximum value of the normal value data is taken as the upper limit of the threshold, the minimum value is taken as the lower limit of the threshold, and the alarm threshold range is taken as the range above the upper limit of the threshold and below the lower limit of the threshold;

[0104] When the index type is a positive type index, the minimum value of the normal value data is taken as the lower limit of the threshold, and the alarm threshold range is taken as the range below the lower limit of the threshold;

[0105] When the index type is a negative type index, the maximum value of the normal value data is taken as the upper limit of the threshold, and the alarm threshold range is taken as the range above the upper limit of the threshold;

[0106] Step S5: extracting real-time index data and an alarm threshold range, and triggering an alarm information when any kind of index data is in the alarm threshold range.

[0107] In a specific embodiment, the monitoring alarm module periodically extracts real-time index data and alarm threshold range from the storage module, and triggers alarm information according to the comparison result of the two.

[0108] When any one of the index results is in the alarm threshold range, the alarm information is triggered.

[0109] The above specific embodiments further explain the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An abnormality alarm method based on an improved abnormality detection algorithm, characterized in that: include: Perform data collection on the indicator source to obtain real-time indicator data, and store historical indicator data and real-time indicator data for threshold calculation; the historical indicator data is a set of N types of indicator data collected from the same indicator source; For each historical indicator data, determine whether it has the same value within the historical time period; If the historical indicator data has the same value within the historical time, determine whether the historical data of the indicator is a normal value based on the manually set threshold A; if the historical data of the indicator is a normal value, determine the indicator type of the historical indicator data, and determine the alarm threshold range of the indicator based on the indicator type; If the historical indicator data has different values ​​within the historical time, the historical indicator data within the threshold A is weighted according to the manually set threshold A, and the historical indicator data outside the threshold A is kept as default; the threshold A includes N thresholds, corresponding to the N types of indicator data in the historical indicator data; Anomaly detection is performed on the historical indicator data based on a weighted anomaly detection algorithm to calculate a threshold value B, output a data anomaly label according to the threshold value B, and obtain normal value data in the historical indicator data; the threshold value B includes N threshold values ​​corresponding to N types of indicator data in the historical indicator data; Determine the indicator type of historical indicator data, obtain the alarm threshold range based on the indicator type and data anomaly label, and store the alarm threshold range; Extract real-time indicator data and alarm threshold ranges. When any indicator data is within the alarm threshold range, an alarm message is triggered.

2. The method according to claim 1, characterized in that The step of weighting the historical indicator data within the threshold A specifically includes: The weighted data is the result of original data*weight.

3. The method according to claim 1, characterized in that The indicator types include: interval indicators, positive indicators, and negative indicators.

4. The method according to claim 1, wherein The method of obtaining and storing an alarm threshold range based on the data anomaly label according to the indicator type specifically includes: When the indicator type is an interval indicator, the maximum value of the normal value data is used as the upper threshold limit, the minimum value is used as the lower threshold limit, and the alarm threshold range is the range above the upper threshold limit and below the lower threshold limit; When the indicator type is a positive indicator, the minimum value of the normal value data is used as the lower limit of the threshold, and the alarm threshold range is the range below the lower limit of the threshold; When the indicator type is a negative indicator, the maximum value of the normal value data is used as the upper threshold limit, and the alarm threshold range is the range above the upper threshold limit.

5. The method according to claim 1, wherein If any of the historical indicator data are the same over a period of time, the manually set threshold A is used to determine whether they are normal values. If they are all outside the threshold A, they are not used and an alarm is triggered. If they are all within the threshold A, the alarm threshold range of the indicator is determined according to the indicator type: When the indicator type is an interval indicator, the alarm threshold range is the range outside the historical indicator data that is the same within the historical time period; When the indicator type is a positive indicator, the alarm threshold range is the range below the same historical indicator data within the historical period; When the indicator type is a negative indicator, the alarm threshold range is a range above the same historical indicator data within the historical time period.

6. A system for the abnormality alarm method according to claim 1, characterized in that: include: The monitoring indicator collection module is used to automatically collect the monitoring indicators of the monitored objects according to the preset collection frequency, collection objects, etc., and transmit them to the data storage module for subsequent analysis; The data storage module is used to store data, on the one hand, it stores the indicator data from the acquisition module, and on the other hand, it stores the alarm threshold range after calculation and analysis for use by the monitoring and alarm module; The task scheduling module is used to automatically schedule task execution based on pre-set task execution information, including extracting historical data from the data storage, calculating thresholds, and storing thresholds in the storage module; The monitoring and alarm module is used to regularly extract the alarm threshold range from the data storage server as the alarm information triggering threshold. At the same time, it regularly extracts real-time indicator data from the data storage module and compares it with the alarm threshold. If it is within the alarm threshold range, an alarm is triggered; The threshold calculation module is used to process the indicator historical data according to the threshold calculation algorithm and the indicator historical data, calculate the threshold corresponding to the indicator, derive the alarm threshold range based on the indicator type, and finally store the alarm threshold range in the output storage module.

Images