Model-based time sequence abnormal index real-time monitoring system and method

Abstract

The invention provides a model-based time sequence anomaly index real-time monitoring device and method, and relates to the field of data anomaly monitoring. The device comprises a data acquisition module for acquiring time sequence data at regular time; the index query module is used for acquiring current and historical time sequence data; the data preprocessing module is used for preprocessing the data; the real-time model prediction module is used for performing model prediction according to the historical time sequence data and calculating a prediction index value; the index calculation module is used for calculating a residual error between an actual index value and a predicted index value of each time sequence, and calculating a standard deviation and a mean deviation based on the residual error; the first judgment module judges whether the index at the current moment is abnormal or not and generates a first abnormal value; the second judgment module is used for inquiring each piece of historical time sequence data in the same period as the current moment and carrying out threshold judgment to identify a second abnormal value; and the abnormity determination module comprehensively judges and finally determines whether abnormity occurs. According to the invention, efficient and accurate anomaly monitoring of the system on the time series data can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data anomaly detection, and in particular to a system and method for real-time monitoring of time series anomaly indicators based on a model. Background Art

[0002] With the development of big data and Internet of Things technologies, the demand for real-time anomaly monitoring of time series data is increasing. Most of the existing open source time series indicator alarm configurations are configured based on static thresholds. In some scenarios, the thresholds are different for different labels or time periods of the same indicator, resulting in underreporting or false alarms. For example, Prometheus is currently widely used for time series indicators, and its abnormal alarm configuration is: by setting a promQL expression and configuring a threshold, promQL is executed regularly to obtain the results of each time series indicator, and an alarm is issued to AlertManager when the threshold is exceeded. Here is an example, such as the message accumulation number kafka_consumergroup_lag of the kafka message queue. There are actually hundreds or thousands of queues, and the thresholds of different queues are different. It is not appropriate to share a threshold to compare abnormalities; and even if it is the same queue, the thresholds for different time periods should not be the same.

[0003] In summary, traditional threshold detection methods can no longer meet the needs of complex and changing business scenarios, and there is an urgent need for a method to efficiently and accurately monitor time series data anomalies. Summary of the invention

[0004] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a system and method for real-time monitoring of time series anomaly indicators based on a model, aiming to achieve efficient and accurate anomaly monitoring of time series data by combining multiple algorithms and data processing technologies.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a real-time monitoring device for time series abnormality indicators based on a model, comprising:

[0006] A data acquisition module is used to regularly collect time series data from a data source;

[0007] An indicator query module, used to obtain current and historical time series data from the data collection module according to a PromQL query statement;

[0008] A data preprocessing module, used to perform preprocessing operations on the acquired data;

[0009] The real-time model prediction module is used to receive preprocessed data, find or initialize the corresponding prediction model for each historical time series; perform model prediction based on historical time series data, and calculate the prediction index value;

[0010] An indicator calculation module, used to calculate the residual between the actual indicator value and the predicted indicator value of each time series, and calculate the standard deviation and mean deviation based on the residual;

[0011] A first judgment module is used to judge whether the indicator at the current moment is abnormal according to the 3σ principle and generate a first abnormal value;

[0012] The second judgment module is used to query each historical time series data of the same period as the current moment and perform threshold judgment to identify the second abnormal value;

[0013] The abnormality determination module is used to make a comprehensive judgment on the first abnormal value and the second abnormal value according to the configured sensitivity or accuracy detection strategy, and finally determine whether they are abnormal.

[0014] Furthermore, the preprocessing operation includes at least one of data cleaning, smoothing, and conversion.

[0015] Furthermore, the real-time model prediction module searches for a corresponding model for each time series, and if no corresponding model is found, a new model is initialized.

[0016] Furthermore, the device also includes: a stationarity check module; the stationarity check module is used to perform a stationarity check on the data processed by the data preprocessing module, and if the stationarity requirement is not met, perform data differential calculation; thereafter, the real-time model prediction module calculates the prediction value of each historical sequence based on the prediction algorithm field, and calculates the final actual indicator prediction value through reverse differential.

[0017] Furthermore, the device also includes:

[0018] Select an algorithm suitable for time series data prediction and train or adjust it according to the data characteristics.

[0019] Furthermore, the indicator calculation module first calculates the predicted indicator value for each of the previous t historical moments and caches it; then calculates the residual of the t moments, that is, residual = actual indicator value - predicted indicator value, and then calculates the standard deviation and mean deviation;

[0020] The first judgment module, according to the 3σ principle, compares whether the residual is greater than [mean deviation ± 3*standard deviation]. If so, it is judged as abnormal, thereby obtaining the first abnormal value.

[0021] Furthermore, the second judgment module is used to query each historical time series data of the same period as the current moment and perform threshold judgment to identify the second abnormal value, including:

[0022] Query the indicators of the most recent t moments in the same period as the current moment, take their minimum and maximum values, and reduce or increase them by 10% as the first threshold, if it exceeds the first threshold, it is counted as a weak anomaly; or, reduce or increase them by 50% as the second threshold, if it exceeds the second threshold, it is counted as a strong anomaly;

[0023] According to the above judgment, the second abnormal value is obtained.

[0024] Furthermore, the device also includes: an indicator alarm module;

[0025] If an abnormality is detected, the indicator alarm module generates alarm information and triggers corresponding response measures according to the configured alarm method.

[0026] Furthermore, the indicator alarm module is secondary developed based on the open source prometheus tool suite, and supports traditional threshold types and automatic detection types; among them, the automatic detection type is used to determine whether it is abnormal in the following way: using the configuration rules as parameters, and regularly adjusting the real-time model detection service interface to obtain the abnormal results of each current sequence.

[0027] In a second aspect, the present invention provides a real-time monitoring method for time series abnormality indicators based on a model, comprising:

[0028] Collect time series data from data sources regularly;

[0029] Obtain current and historical time series data based on PromQL query statements;

[0030] Perform preprocessing operations on the acquired data;

[0031] Receive preprocessed data, find or initialize the corresponding prediction model for each historical time series; perform model prediction based on historical time series data and calculate the prediction index value;

[0032] Calculate the residuals between the actual indicator values ​​and the predicted indicator values ​​for each time series, and calculate the standard deviation and mean deviation based on the residuals;

[0033] According to the 3σ principle, determine whether the indicator at the current moment is abnormal and generate the first abnormal value;

[0034] Query each historical time series data of the same period as the current moment and perform threshold judgment to identify the second abnormal value;

[0035] According to the configured sensitivity or accuracy detection strategy, a comprehensive judgment is made on the first abnormal value and the second abnormal value to finally determine whether they are abnormal.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention realizes efficient and accurate anomaly monitoring of time series data by a general alarm system by combining multiple algorithms and data processing technologies. The device and method have the advantages of strong real-time performance, high accuracy, and good flexibility, and can be widely used in various business scenarios that require real-time anomaly monitoring. For some scenarios where traditional threshold methods are not suitable, such as the number of queue message accumulations, the number of requests, etc., the model recognition method is used to automatically identify anomalies based on historical and contemporaneous data, which can more accurately and in real time identify potential problems or abnormal behaviors and reduce missed reports. The indicator alarm module can monitor the health status of the system more intelligently and accurately, and promptly notify relevant personnel before or when a problem occurs, thereby improving the stability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of a module of a device for real-time monitoring of abnormal time series indicators based on a model provided by an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of anomaly detection logic according to an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of the prediction processing logic of an embodiment of the present invention;

[0041] Figure 4 A flow chart of a method for real-time monitoring of time series abnormality indicators based on a model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Figure 1 The present invention provides a module schematic diagram of a real-time monitoring device for time series abnormality indicators based on a model. Figures 1 to 3 As shown, a real-time monitoring device 100 for time series abnormality indicators based on a model is applied to a general alarm system, comprising:

[0044] The data collection module 110 is used to collect time series data from a data source (such as Prometheus) at regular intervals;

[0045] The indicator query module 120 is used to obtain current and historical time series data from the data acquisition module according to the PromQL query statement; for example, the current indicator value can be queried regularly through PromQL to ensure the real-time nature of the data.

[0046] The data preprocessing module 130 is used to perform preprocessing operations on the acquired data; specifically, the preprocessing operations include: performing one or more preprocessing operations such as cleaning, smoothing, and conversion on the original data to improve data quality.

[0047] The real-time model prediction module 140 is used to receive the pre-processed data, search for or initialize the corresponding prediction model for each historical time series; perform model prediction based on the historical time series data, and calculate the prediction index value;

[0048] An indicator calculation module 150 is used to calculate the residual between the actual indicator value and the predicted indicator value of each time series, and calculate the standard deviation and the mean deviation based on the residual;

[0049] A first judgment module 160 is used to judge whether the indicator at the current moment is abnormal according to the 3σ principle, and generate a first abnormal value;

[0050] The second judgment module 170 is used to query each historical time series data of the same period as the current moment and perform threshold judgment to identify the second abnormal value;

[0051] The anomaly determination module 180 is used to make a comprehensive judgment on the first anomaly value and the second anomaly value according to the configured sensitivity (preferring a false alarm to miss a possible anomaly) or accuracy detection strategy (trying to report as little as possible to ensure that the reported anomaly is a true anomaly), and finally determine whether it is an anomaly.

[0052] According to the above-mentioned embodiment of the present invention, the device uses the present model recognition method to automatically identify anomalies based on historical and contemporaneous data, and can more accurately and real-time identify potential problems or abnormal behaviors, thereby reducing missed reports.

[0053] Furthermore, the real-time model prediction module 140 searches for a corresponding model for each time series, and if no corresponding model is found, a new model is initialized.

[0054] Furthermore, in order to ensure the stationarity of the data series, Figure 3 As shown, the device 100 further includes: a stability inspection module;

[0055] The stationarity test module is used to perform a stationarity test on the data processed by the data preprocessing module 130. If the stationarity requirement is not met, data differential calculation is performed; then, the real-time model prediction module 140 calculates the prediction value of each historical sequence based on the prediction algorithm field, and calculates the final actual indicator prediction value through reverse differential.

[0056] Furthermore, the device 100 selects an algorithm suitable for time series data prediction (such as ARIMA, LSTM, etc.) and performs training or adjustment according to data characteristics.

[0057] Furthermore, the indicator calculation module 150 first calculates the predicted indicator value for each of the previous t historical moments and caches it; then calculates the residual of the t moments, that is, residual = actual indicator value - predicted indicator value, and then calculates the standard deviation and mean deviation;

[0058] In actual use, in order to reduce the impact of glitches, the time series value at time t generally chooses the average value, such as the average of the previous minute. For scenarios with sudden increases or decreases, the trend recognition based on the difference between the actual index and the predicted index can reach more than 50%; for scenarios with slow increases or decreases, based on the comparison of the threshold of historical data in the same period, it can also reach more than 50%.

[0059] The first judgment module 160, according to the 3σ principle, compares whether the residual is greater than [mean deviation ± 3*standard deviation]. If so, it is judged as abnormal, thereby obtaining a first abnormal value.

[0060] The second judgment module 170 is used to query each historical time series data of the same period as the current moment and perform threshold judgment to identify the second abnormal value, including:

[0061] Query the indicators of the most recent t moments in the same period as the current moment, take their minimum and maximum values, and reduce or increase them by 10% as the first threshold, if it exceeds the first threshold, it is counted as a weak anomaly; or, reduce or increase them by 50% as the second threshold, if it exceeds the second threshold, it is counted as a strong anomaly;

[0062] According to the above judgment, the second abnormal value is obtained.

[0063] Furthermore, the apparatus 100 further includes: an indicator alarm module;

[0064] The indicator alarm module, if an abnormality is detected, generates alarm information and triggers corresponding response measures according to the configured alarm mode. It allows users to configure alarm strategies and thresholds according to business needs, supports configuration of alarm types and response measures according to business needs, supports dynamic adjustment of alarm strategies, and realizes personalized customization of alarms to adapt to the ever-changing business environment.

[0065] Furthermore, the indicator alarm module is secondary developed based on the open source prometheus tool suite, and supports traditional threshold types and automatic detection types. Among them, the automatic detection type is used to determine whether it is abnormal by taking the configuration rules as parameters and regularly calling the real-time model detection service interface to obtain the abnormal results of each current sequence.

[0066] Among them, the traditional threshold type: This is the most common alarm method, which uses the preset threshold to determine whether a certain indicator exceeds the normal range, thereby triggering an alarm. For example, if the CPU usage exceeds 80%, an alarm notification is sent.

[0067] Automatic detection type: This method is more intelligent and flexible, and it does not rely solely on static thresholds. Instead, it allows users to define more complex rules and dynamically determine whether an indicator is abnormal by calling the real-time model detection service interface. For example, in the present invention, these rules can include time series analysis, machine learning model prediction and other technologies to more accurately identify potential problems or abnormal behaviors.

[0068] In the automatic detection type of the present invention, the indicator alarm module will send the configured rules as parameters to the real-time model detection service interface at regular intervals (such as every minute or every hour). This interface will perform real-time analysis on the currently monitored sequence (such as CPU usage, memory usage, etc.) according to the provided rules, and return the abnormal results of each sequence. If a sequence is judged to be abnormal, the alarm module will send the corresponding alarm notification according to the configuration, which may include multiple forms such as email, text message, Slack message, etc.

[0069] In this way, the indicator alarm module can monitor the health of the system more intelligently and accurately, and notify relevant personnel in time before or when problems occur, thereby improving the stability and reliability of the system.

[0070] The above is an introduction to the device embodiment. The following is a method embodiment to further illustrate the solution of the present invention.

[0071] Figure 4 The flowchart of a method 200 for real-time monitoring of time series abnormality indicators based on a model provided by an embodiment of the present invention is as follows. Figure 4 As shown, a model-based real-time monitoring method 200 for time series abnormality indicators includes:

[0072] S210: Collecting time series data from the data source at regular intervals;

[0073] S220: Obtain current and historical time series data according to the PromQL query statement;

[0074] S230: performing preprocessing operations on the acquired data;

[0075] S240: receiving the preprocessed data, searching for or initializing a corresponding prediction model for each historical time series; performing model prediction based on the historical time series data, and calculating a prediction index value;

[0076] S250: Calculate the residual between the actual index value and the predicted index value of each time series, and calculate the standard deviation and the mean deviation based on the residual;

[0077] S260: judging whether the indicator at the current moment is abnormal according to the 3σ principle, and generating a first abnormal value;

[0078] S270: query each historical time series data of the same period as the current moment and perform threshold judgment to identify the second abnormal value;

[0079] S280: According to the configured sensitivity or accuracy detection strategy, a comprehensive judgment is made on the first abnormal value and the second abnormal value to finally determine whether they are abnormal.

[0080] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0081] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A real-time monitoring device for time series abnormality indicators based on a model, characterized in that: include: A data acquisition module is used to regularly collect time series data from a data source; An indicator query module, used to obtain current and historical time series data from the data collection module according to a PromQL query statement; A data preprocessing module, used to perform preprocessing operations on the acquired data; The real-time model prediction module is used to receive preprocessed data, find or initialize the corresponding prediction model for each historical time series; perform model prediction based on historical time series data, and calculate the prediction index value; An indicator calculation module, used to calculate the residual between the actual indicator value and the predicted indicator value of each time series, and calculate the standard deviation and mean deviation based on the residual; A first judgment module is used to judge whether the indicator at the current moment is abnormal according to the 3σ principle and generate a first abnormal value; The second judgment module is used to query each historical time series data of the same period as the current moment and perform threshold judgment to identify the second abnormal value; The abnormality determination module is used to make a comprehensive judgment on the first abnormal value and the second abnormal value according to the configured sensitivity or accuracy detection strategy, and finally determine whether they are abnormal.

2. The device according to claim 1, characterized in that in, The preprocessing operation includes at least one of data cleaning, smoothing, and conversion.

3. The device according to claim 1, characterized in that The real-time model prediction module searches for a corresponding model for each time series, and if no corresponding model is found, a new model is initialized.

4. The device according to claim 1, characterized in that The device also includes: a stability testing module; The stationarity check module is used to perform a stationarity check on the data processed by the data preprocessing module. If the stationarity requirement is not met, data differential calculation is performed; thereafter, the real-time model prediction module calculates the prediction value of each historical sequence based on the prediction algorithm field, and calculates the final actual indicator prediction value through reverse differential calculation.

5. The device according to claim 4, characterized in that The device also includes: Select an algorithm suitable for time series data prediction and train or adjust it according to the data characteristics.

6. The device according to claim 1, characterized in that in, The indicator calculation module first calculates the predicted indicator value for each of the previous t historical moments and caches it; then calculates the residual of the t moments, that is, residual = actual indicator value - predicted indicator value, and then calculates the standard deviation and mean deviation; The first judgment module, according to the 3σ principle, compares whether the residual is greater than [mean deviation ± 3*standard deviation]. If so, it is judged as abnormal, thereby obtaining the first abnormal value.

7. The device according to claim 6, characterized in that The second judgment module is used to query each historical time series data of the same period as the current moment and perform threshold judgment to identify the second abnormal value, including: Query the indicators of the most recent t moments in the same period as the current moment, take their minimum and maximum values, and reduce or increase them by 10% as the first threshold, if it exceeds the first threshold, it is counted as a weak anomaly; or, reduce or increase them by 50% as the second threshold, if it exceeds the second threshold, it is counted as a strong anomaly; According to the above judgment, the second abnormal value is obtained.

8. The device according to claim 1, characterized in that Also includes: Indicator alarm module; If an abnormality is detected, the indicator alarm module generates alarm information and triggers corresponding response measures according to the configured alarm method.

9. The device according to claim 8, characterized in that The indicator alarm module is secondary developed based on the open source prometheus tool suite, and supports traditional threshold types and automatic detection types; the automatic detection type is used to determine whether it is abnormal in the following way: using the configuration rules as parameters and regularly adjusting the real-time model detection service interface to obtain the abnormal results of each current sequence.

10. A real-time monitoring method for time series abnormality indicators based on a model, characterized in that: include: Collect time series data from data sources regularly; Obtain current and historical time series data based on PromQL query statements; Perform preprocessing operations on the acquired data; Receive preprocessed data, search or initialize the corresponding prediction model for each historical time series; perform model prediction based on historical time series data and calculate the prediction index value; Calculate the residuals between the actual indicator values ​​and the predicted indicator values ​​for each time series, and calculate the standard deviation and mean deviation based on the residuals; According to the 3σ principle, determine whether the indicator at the current moment is abnormal and generate the first abnormal value; Query each historical time series data of the same period as the current moment and perform threshold judgment to identify the second abnormal value; According to the configured sensitivity or accuracy detection strategy, a comprehensive judgment is made on the first abnormal value and the second abnormal value to finally determine whether they are abnormal.

Images