Self-adaptive data abnormal value detection method and terminal

Through the adaptive data outlier detection method, combined with time series decomposition and dynamic threshold adjustment, the problem of threshold fixed in the prior art is solved, the detection accuracy and efficiency are improved, and the efficient operation of the system is achieved.

CN119004320APending Publication Date: 2024-11-22FUJIAN WANFU INFORMATION TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411070960.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing data outlier detection methods rely on fixed threshold settings and cannot adapt to changes in data distribution, resulting in low detection accuracy and efficiency, and large calculation volume affects system operation.

Method used

Adaptive data outlier detection method is adopted to perform initial screening by receiving data in real time, and timely pulling historical data for timed decomposition, determine trends, seasonalities and residuals, dynamically adjust the threshold range, and use appropriate time series models for Test.

Benefits of technology

It improves the accuracy and efficiency of data outliers detection, reduces the need for artificial adjustment of thresholds, the system has better generalization capabilities, the real-time calculation part is small, and timed calculations can detect complex anomalies without affecting the normal operation of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004320A_ABST
    Figure CN119004320A_ABST
Patent Text Reader

Abstract

The invention provides a self-adaptive data abnormal value detection method and a terminal. The self-adaptive data abnormal value detection method comprises the following steps: receiving data in real time, and screening according to an initial threshold range to obtain primarily screened data; the method comprises the following steps: regularly pulling historical data, and decomposing the historical data based on a time sequence decomposition method to obtain a trend, seasonality and a residual error of the historical data; determining an adopted time sequence model based on the trend, the seasonality and the residual error; and detecting the preliminary screening data based on the time sequence model, determining an abnormal value, and dynamically correcting the initial threshold range according to the abnormal value. According to the invention, the threshold range can be dynamically adjusted according to the actual condition of the data, and the detection accuracy and detection efficiency of the data abnormal value are effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an adaptive data outlier detection method and terminal. Background Art

[0002] Data outlier detection is a key function in IoT systems, which is responsible for ensuring the accuracy, completeness and reliability of data collected from various IoT devices. This function monitors and analyzes data in real time, identifies and corrects errors or anomalies in the data, and ensures that the data quality meets the system requirements and standards.

[0003] The usual method of detecting abnormal data is to configure the threshold, set the data value range, or the percentage of data change, and then calculate and determine whether it is an abnormal value when receiving the data. This method has the advantages of being simple and easy to implement, good real-time performance, and strong interpretability, but it still has the following defects:

[0004] Disadvantage 1: Accuracy depends on the setting of thresholds, which needs to be based on a deep understanding of data characteristics. If the thresholds are set improperly, the detection effect may be affected.

[0005] Disadvantage 2: Lack of flexibility. The threshold is fixed and cannot adapt to changes in data distribution. In complex scenarios, threshold detection may not be able to effectively identify data that does not obviously exceed the threshold but is still abnormal.

[0006] Disadvantage 3: The above calculations are performed while receiving data. If the amount of data involved in the calculation is large, it will be time-consuming and easily affect the normal operation of the system; if the amount of data involved in the calculation is small, it will easily cause false alarms and missed alarms. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide an adaptive data outlier detection method and terminal, which can dynamically adjust the threshold range according to the actual data situation, and effectively improve the detection accuracy and efficiency of data outliers.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] An adaptive data outlier detection method comprises the steps of:

[0010] S1, receiving data in real time and screening according to the initial threshold range to obtain preliminary screening data;

[0011] S2. Periodically pull historical data, and decompose the historical data based on a time series decomposition method to obtain trend information, seasonal information, and residuals of the historical data;

[0012] S3. Determine a time series model to be used based on the trend information, the seasonal information and the residual;

[0013] S4. Detect the initial screening data based on the time series model, determine abnormal values, and dynamically correct the initial threshold range according to the abnormal values.

[0014] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0015] An adaptive data outlier detection terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0016] S1, receiving data in real time and screening according to the initial threshold range to obtain preliminary screening data;

[0017] S2. Periodically pull historical data, and decompose the historical data based on a time series decomposition method to obtain the trend, seasonality, and residual of the historical data;

[0018] S3. Determine a time series model to be used based on the trend, the seasonality and the residual;

[0019] S4. Detect the initial screening data based on the time series model, determine abnormal values, and dynamically correct the initial threshold range according to the abnormal values.

[0020] The beneficial effects of the present invention are: providing an adaptive data outlier detection method and terminal, by preliminarily screening the real-time received data according to the initial threshold range to remove obvious abnormal data, combining the trend, seasonality and residual obtained by time series analysis of the historical data pulled at regular intervals, so as to determine the time series model to be used for further outlier screening of the preliminarily screened data, and after obtaining accurate outliers, reversely adjusting the initial threshold range used for the preliminarily screened abnormal data, thereby realizing dynamic adjustment of the threshold range, not relying on manual regular adjustment, and making the threshold range more and more accurate with the continuous acquisition of data and the continuous selection of models, and cooperating with the method of regularly analyzing historical data and selecting a suitable model for detection, so that the system has better generalization ability, effectively improving the accuracy of outlier detection, and the overall calculation is divided into real-time calculation and scheduled calculation, wherein the real-time calculation part has a small amount of calculation, but can filter obvious abnormal data, and the scheduled calculation will apply the model to analyze a large amount of data, can find complex anomalies, can be performed when the system is idle, does not affect the normal system operation, and effectively improves the detection efficiency of data outliers. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1This is an overall flow chart of an adaptive data outlier detection method according to an embodiment of the present invention;

[0022] Figure 2 A specific process framework of an adaptive data outlier detection method according to an embodiment of the present invention;

[0023] Figure 3 This is a specific flow chart of step S3 in an adaptive data outlier detection method according to an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of the structure of an adaptive data outlier detection terminal according to an embodiment of the present invention.

[0025] Description of labels:

[0026] 1. An adaptive data outlier detection terminal; 2. A memory; 3. A processor. DETAILED DESCRIPTION

[0027] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in conjunction with the implementation modes and the accompanying drawings.

[0028] Prior to this, the English terms that appear in this article are explained as follows:

[0029] Loess smoothing algorithm: It is a local smoothing method in non-parametric statistics. It smoothes the data features by locally weighting and fitting the data, so as to better capture the laws of the data.

[0030] Please refer to Figures 1 to 3 , an adaptive data outlier detection method, comprising the steps of:

[0031] S1, receiving data in real time and screening according to the initial threshold range to obtain preliminary screening data;

[0032] S2. Periodically pull historical data, and decompose the historical data based on a time series decomposition method to obtain the trend, seasonality, and residual of the historical data;

[0033] S3. Determine a time series model to be used based on the trend, the seasonality and the residual;

[0034] S4. Detect the initial screening data based on the time series model, determine abnormal values, and dynamically correct the initial threshold range according to the abnormal values.

[0035] From the above description, it can be seen that the beneficial effects of the present invention are: providing an adaptive data outlier detection method, by preliminarily screening the real-time received data according to the initial threshold range to remove obvious abnormal data, combining the trend, seasonality and residual obtained by time series analysis of historical data pulled at regular intervals to determine the time series model used to further screen the data after the initial screening, and after obtaining accurate outliers, reversely adjust the initial threshold range used for the initial screening abnormal data, thereby realizing dynamic adjustment of the threshold range, without relying on manual regular adjustment, and making the threshold range more and more accurate with the continuous acquisition of data and the continuous selection of models, and cooperating with the method of regularly analyzing historical data and selecting a suitable model for detection, so that the system has better generalization ability, effectively improving the accuracy of outlier detection, and the overall calculation is divided into real-time calculation and scheduled calculation, among which the real-time calculation part has a small amount of calculation, but can filter obvious abnormal data, and the scheduled calculation will apply the model to analyze a large amount of data, can find complex anomalies, can be performed when the system is idle, does not affect the normal system operation, and effectively improves the detection efficiency of data outliers.

[0036] Furthermore, the step S2 is specifically as follows:

[0037] S21, regularly query the historical data list and pull the historical data;

[0038] S22, based on the seasonal time series decomposition method, using the Loess smoothing algorithm to smooth the original time series of the historical data to obtain initial trend information;

[0039] S23, subtracting the initial trend information from the original time series to obtain the detrended historical data;

[0040] S24, using the Loess smoothing algorithm again to smooth the time series of the historical data after detrending to obtain initial seasonal information;

[0041] S25, subtracting the initial seasonal information from the original time series to obtain the de-seasonalized historical data;

[0042] S26, using the Loess smoothing algorithm again to smooth the time series of the deseasonalized historical data, and updating the initial trend information;

[0043] S27, repeating steps S23-S26 until the trend information and the seasonal information converge to obtain final trend information and final seasonal information;

[0044] S28, calculating the variance and residual of the final trend information and the variance and residual of the final seasonal information to obtain trend variance, trend residual, seasonal variance and seasonal residual;

[0045] S29. Calculate the trend strength and seasonal strength of the historical data based on the trend variance, the trend residual, the seasonal variance and the seasonal residual.

[0046] From the above description, it can be seen that the seasonal time series decomposition method can handle nonlinear trends and variable seasonal patterns, and is highly robust to outliers, effectively ensuring the accurate detection of outliers; the Loess smoothing algorithm can fit low-order polynomials in local areas to estimate trend information, which helps to capture the long-term change trend of historical data while smoothing short-term fluctuations; and it can capture periodic models in different local areas to estimate seasonal information.

[0047] Furthermore, the step S29 is specifically as follows:

[0048] Calculating the ratio of the trend variance to the variance of the trend residual to obtain the trend strength of the historical data;

[0049] The ratio of the seasonal variance to the variance of the seasonal residual is calculated to obtain the seasonal intensity of the historical data.

[0050] From the above description, we can see that after the historical data is decomposed by the seasonal time series decomposition method, the trend strength and seasonal strength can be measured by comparing the variance of the trend information and seasonal information with the variance of the residual. This can effectively judge the trend and seasonality in the time series and facilitate the subsequent detection model selection.

[0051] Furthermore, the time series model includes a seasonal prediction model, a trend prediction model and a smoothing prediction model;

[0052] The step S3 is specifically as follows:

[0053] S31, if the seasonal intensity is greater than a first threshold, select the seasonal prediction model, otherwise proceed to step S32;

[0054] S32. If the trend strength is greater than the first threshold, select the trend prediction model; otherwise, select the smooth prediction model.

[0055] From the above description, it can be seen that the ratio of the trend variance to the variance of the trend residual, as well as the ratio of the seasonal variance to the variance of the seasonal residual are determined. These ratios represent the relative importance of trend information and seasonal information in the entire time series of the pulled historical data. If the ratio is close to 1, it means that the information dominates the time series. If the ratio is close to 0, it means that the information is not significant in the time series.

[0056] Furthermore, the historical data pulled at each time in step S2 is the historical data within a preset recent period;

[0057] The seasonal prediction model includes a seasonal multi-source regression model, a seasonal autoregression model and a time series decomposition model;

[0058] The trend prediction model includes a linear trend prediction model, a nonlinear trend prediction model and an autoregressive prediction model;

[0059] The smoothing prediction model includes simple smoothing method, moving smoothing method and exponential smoothing method.

[0060] From the above description, we can see that since historical data is constantly accumulating and changing, if all historical data are used as data for analyzing trends and seasonality, not only will the amount of calculation be large, but the trend and seasonality judgment of the overall historical data will also be vague. Therefore, by presetting the most recent time period to regularly pull recent historical data instead of all historical data, the amount of calculation can be effectively reduced while ensuring the accuracy of model selection, further improving the detection accuracy and efficiency of outliers in the data outlier detection process.

[0061] Please refer to Figure 4 , an adaptive data outlier detection terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0062] S1, receiving data in real time and screening according to the initial threshold range to obtain preliminary screening data;

[0063] S2. Periodically pull historical data, and decompose the historical data based on a time series decomposition method to obtain the trend, seasonality, and residual of the historical data;

[0064] S3. Determine a time series model to be used based on the trend, the seasonality and the residual;

[0065] S4. Detect the initial screening data based on the time series model, determine abnormal values, and dynamically correct the initial threshold range according to the abnormal values.

[0066] From the above description, it can be seen that the beneficial effect of the present invention is that: based on the same technical concept, in conjunction with the above-mentioned adaptive data outlier detection method, an adaptive data outlier detection terminal is provided, by preliminarily screening the real-time received data according to the initial threshold range to remove obvious abnormal data, combining the trend, seasonality and residual obtained by time series analysis of the time-series data to determine the time series model used to further screen the data after the initial screening, and after obtaining the accurate outlier, the initial threshold range used for the initial screening abnormal data is adjusted in reverse, thereby realizing dynamic adjustment of the threshold range, without relying on manual regular adjustment, and the threshold range can be made more and more accurate with the continuous acquisition of data and the continuous selection of the model, and in conjunction with the method of regularly analyzing historical data and selecting a suitable model for detection, the system has better generalization ability, effectively improving the accuracy of outlier detection, and the overall calculation is divided into real-time calculation and scheduled calculation, wherein the real-time calculation part has a small amount of calculation, but can filter obvious abnormal data, and the scheduled calculation will apply the model to analyze a large amount of data, can find complex anomalies, can be performed when the system is idle, does not affect the normal system operation, and effectively improves the detection efficiency of data outliers.

[0067] Furthermore, the step S2 is specifically as follows:

[0068] S21, regularly query the historical data list and pull the historical data;

[0069] S22, based on the seasonal time series decomposition method, using the Loess smoothing algorithm to smooth the original time series of the historical data to obtain initial trend information;

[0070] S23, subtracting the initial trend information from the original time series to obtain the detrended historical data;

[0071] S24, using the Loess smoothing algorithm again to smooth the time series of the historical data after detrending to obtain initial seasonal information;

[0072] S25, subtracting the initial seasonal information from the original time series to obtain the de-seasonalized historical data;

[0073] S26, using the Loess smoothing algorithm again to smooth the time series of the deseasonalized historical data, and updating the initial trend information;

[0074] S27, repeating steps S23-S26 until the trend information and the seasonal information converge to obtain final trend information and final seasonal information;

[0075] S28, calculating the variance and residual of the final trend information and the variance and residual of the final seasonal information to obtain trend variance, trend residual, seasonal variance and seasonal residual;

[0076] S29. Calculate the trend strength and seasonal strength of the historical data based on the trend variance, the trend residual, the seasonal variance and the seasonal residual.

[0077] From the above description, it can be seen that the seasonal time series decomposition method can handle nonlinear trends and variable seasonal patterns, and is highly robust to outliers, effectively ensuring the accurate detection of outliers; the Loess smoothing algorithm can fit low-order polynomials in local areas to estimate trend information, which helps to capture the long-term change trend of historical data while smoothing short-term fluctuations; and it can capture periodic models in different local areas to estimate seasonal information.

[0078] Furthermore, the step S29 is specifically as follows:

[0079] Calculating the ratio of the trend variance to the variance of the trend residual to obtain the trend strength of the historical data;

[0080] The ratio of the seasonal variance to the variance of the seasonal residual is calculated to obtain the seasonal intensity of the historical data.

[0081] From the above description, it can be seen that after the historical data is decomposed by the seasonal time series decomposition method, the trend strength and seasonal strength can be measured by comparing the variance of the trend information and seasonal information with the variance of the residual, that is, by determining the ratio of the trend variance to the variance of the trend residual, and the ratio of the seasonal variance to the variance of the seasonal residual. These ratios represent the relative importance of the trend information and seasonal information in the entire time series of the pulled historical data. If the ratio is close to 1, it means that the information dominates the time series. If the ratio is close to 0, it means that the information is not significant in the time series, which is convenient for the subsequent detection model selection.

[0082] Furthermore, the time series model includes a seasonal prediction model, a trend prediction model and a smoothing prediction model;

[0083] The step S3 is specifically as follows:

[0084] S31, if the seasonal intensity is greater than a first threshold, select the seasonal prediction model, otherwise proceed to step S32;

[0085] S32. If the trend strength is greater than the first threshold, select the trend prediction model; otherwise, select the smooth prediction model.

[0086] From the above description, it can be seen that the ratio of the trend variance to the variance of the trend residual, as well as the ratio of the seasonal variance to the variance of the seasonal residual are determined. These ratios represent the relative importance of trend information and seasonal information in the entire time series of the pulled historical data. If the ratio is close to 1, it means that the information dominates the time series. If the ratio is close to 0, it means that the information is not significant in the time series.

[0087] Furthermore, the historical data pulled at each time in step S2 is the historical data within a preset recent period;

[0088] The seasonal prediction model includes a seasonal multi-source regression model, a seasonal autoregression model and a time series decomposition model;

[0089] The trend prediction model includes a linear trend prediction model, a nonlinear trend prediction model and an autoregressive prediction model;

[0090] The smoothing prediction model includes simple smoothing method, moving smoothing method and exponential smoothing method.

[0091] From the above description, we can see that since historical data is constantly accumulating and changing, if all historical data are used as data for analyzing trends and seasonality, not only will the amount of calculation be large, but the trend and seasonality judgment of the overall historical data will also be vague. Therefore, by presetting the most recent time period to regularly pull recent historical data instead of all historical data, the amount of calculation can be effectively reduced while ensuring the accuracy of model selection, further improving the detection accuracy and efficiency of outliers in the data outlier detection process.

[0092] The present invention provides an adaptive data outlier detection method and terminal, which are mainly used in data quality monitoring scenarios, and timely detect outliers in the data for processing, thereby ensuring data quality. The following is a specific description in conjunction with specific embodiments:

[0093] Please refer to Figure 1 , Embodiment 1 of the present invention is:

[0094] An adaptive data outlier detection method, such as Figure 1 As shown, the steps include:

[0095] S1. Receive data in real time and filter according to the initial threshold range to obtain preliminary screening data.

[0096] The initial threshold range may be set in advance by the staff based on standards and experience.

[0097] S2. Periodically pull historical data and decompose it based on the time series decomposition method to obtain the trend, seasonality and residual of the historical data.

[0098] S3. Determine the time series model to be used based on trend, seasonality and residuals.

[0099] S4. Detect the initial screening data based on the time series model, determine the outliers, and dynamically correct the initial threshold range based on the outliers.

[0100] It is worth mentioning that the time series model is used to analyze time series data and can directly return whether a data value is an outlier. For example, we have trained an SVM model for finding abnormal water level data and input a set of water level data into the SVM model. The model will return 1 or -1, where 1 represents normal and -1 represents abnormal.

[0101] That is, in this embodiment, the data received in real time is initially screened according to the initial threshold range to remove obvious abnormal data, and the trend, seasonality and residual obtained by time series analysis of historical data is combined to determine the time series model to be used for further outlier screening of the data after the initial screening. After obtaining accurate outliers, the initial threshold range used for the initial screening of abnormal data is adjusted in reverse, thereby realizing dynamic adjustment of the threshold range, without relying on manual regular adjustment. The threshold range can be made more and more accurate with the continuous acquisition of data and the continuous selection of models, and the system has better generalization ability by coordinating the analysis of historical data and selecting the appropriate model for detection. While effectively improving the accuracy of outlier detection, the overall calculation is divided into real-time calculation and scheduled calculation, wherein the real-time calculation part has a small amount of calculation, but can filter obvious abnormal data, and the scheduled calculation will apply the model to analyze a large amount of data, can find complex anomalies, can be performed when the system is idle, does not affect the normal system operation, and effectively improves the detection efficiency of data outliers.

[0102] Please refer to Figures 2 to 3 , Embodiment 2 of the present invention is:

[0103] An adaptive data outlier detection method, based on the above embodiment 1, such as Figure 2 As shown, in this embodiment, step S2 is specifically as follows:

[0104] S21. Periodically query the historical data list and pull historical data.

[0105] At the same time, the operating parameters configured by the background of the system can also be detected regularly, including the execution time, frequency, and data range of the detection. At the same time, in this embodiment, since historical data is constantly accumulated and changed, if all historical data are used as data for analyzing trends and seasonality, not only will the amount of calculation be large, but the trend and seasonality judgment of the overall historical data will also be general. Therefore, the historical data pulled regularly each time in step S2 can be the historical data within the preset recent time period, that is, the recent historical data is pulled regularly by the preset recent time period instead of all historical data, which effectively reduces the amount of calculation while ensuring the accuracy of model selection, and further improves the detection accuracy and efficiency of outliers in the process of data outlier detection.

[0106] After pulling the historical data, it is necessary to analyze whether its time series has trends and periodicity. The method used in this embodiment is the seasonal time series decomposition method STL, which can handle nonlinear trends and variable seasonal patterns by decomposing the time series into components such as trends, seasonality and random fluctuations. At the same time, it has strong robustness to outliers, effectively ensuring the accurate detection of outliers.

[0107] The detailed implementation is as follows:

[0108] Seasonal time series decomposition method STL (Seasonal and Trend decomposition using Loess) is a powerful decomposition method used to decompose time series data into three main information: trend, seasonality and residual. It uses local weighted regression (Loess) for smoothing. The Loess smoothing algorithm is a non-parametric method that is applicable to many types of time series data.

[0109] Here is an explanation of how STL detects seasonality and trend:

[0110] 1. Trend

[0111] Trend information represents the long-term direction or pattern in a time series. STL uses the Loess smoothing algorithm to estimate trend information by fitting a low-order polynomial in a local area. This process helps capture the long-term trend of the data while smoothing short-term fluctuations.

[0112] 2. Seasonal

[0113] Seasonal information represents a periodic fluctuation pattern in a time series, which usually repeats at fixed time intervals. STL estimates seasonal information by using the Loess smoothing algorithm after subtracting the trend information from the time series data. This process can capture periodic patterns in different local areas.

[0114] 3. Residual

[0115] Residual information is the part of the time series that remains after removing trend and seasonal information. These residuals usually represent random fluctuations or noise in the data.

[0116] In this embodiment, the decomposition steps of STL are as follows:

[0117] S22. Based on the seasonal time series decomposition method, the Loess smoothing algorithm is used to smooth the original time series of historical data to obtain the initial trend information.

[0118] S23. Subtract the initial trend information from the original time series to obtain the detrended historical data.

[0119] S24. Use the Loess smoothing algorithm again to smooth the time series of the detrended historical data to obtain the initial seasonal information.

[0120] S25. Subtract the initial seasonal information from the original time series to obtain deseasonalized historical data.

[0121] S26. Use the Loess smoothing algorithm again to smooth the time series of de-seasonalized historical data and update the initial trend information.

[0122] S27. Repeat steps S23-S26 until the trend information and the seasonal information converge to obtain final trend information and final seasonal information.

[0123] Then we need to judge the strength of trend information and seasonal information. The steps are as follows:

[0124] S28. Calculate the variance and residual of the final trend information and the variance and residual of the final seasonal information to obtain the trend variance, trend residual, seasonal variance and seasonal residual.

[0125] S29. The trend strength and seasonal strength of the historical data are calculated based on the trend variance, trend residual, seasonal variance and seasonal residual. Specifically,

[0126] The ratio of the trend variance to the variance of the trend residual is calculated to obtain the trend strength of the historical data;

[0127] The ratio of the seasonal variance to the variance of the seasonal residual is calculated to obtain the seasonal intensity of the historical data.

[0128] That is, after the historical data are decomposed by the seasonal time series decomposition method, the trend strength and seasonality strength can be measured by comparing the variance of the trend information and seasonal information with the variance of the residual. This can effectively judge the trend and seasonality in the time series and facilitate the subsequent detection model selection.

[0129] In this embodiment, the time series model includes a seasonal prediction model, a trend prediction model and a smooth prediction model. Figure 3 As shown, step S3 is specifically as follows:

[0130] S31. If the seasonal intensity is greater than the first threshold, select the seasonal prediction model, otherwise go to step S32.

[0131] S32. If the trend strength is greater than the first threshold, select the trend prediction model; otherwise, select the smoothing prediction model.

[0132] Among them, the seasonal prediction models include seasonal multi-source regression model, seasonal autoregression model and time series decomposition model; the trend prediction models include linear trend inference model, nonlinear trend inference model and autoregression prediction model; the smoothing prediction models include simple smoothing method, moving smoothing method and exponential smoothing method.

[0133] That is, the ratio of the trend variance to the variance of the trend residual, as well as the ratio of the seasonal variance to the variance of the seasonal residual are determined. These ratios represent the relative importance of trend information and seasonal information in the entire time series of the pulled historical data. If the ratio is close to 1, it means that the information dominates the time series. If the ratio is close to 0, it means that the information is not significant in the time series.

[0134] In this embodiment, the first threshold can be set to 0.5. When the seasonal intensity is greater than 0.5, the time series is considered to have significant seasonal information, and a seasonal prediction model can be selected; when the seasonal intensity is less than or equal to 0.5, but the trend intensity is greater than 0.5, the time series is considered to have significant trend information, and a trend prediction model can be selected; when both the seasonal intensity and the trend intensity are less than or equal to 0.5, it means that both the seasonal information and the trend information of the time series are significant, and only an ordinary smoothing prediction method is required. In this embodiment, seasonality is periodicity.

[0135] Please refer to Figure 4 , Embodiment 3 of the present invention is:

[0136] An adaptive data outlier detection terminal 1 includes a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3. When the processor 3 executes the computer program, the steps in an adaptive data outlier detection method in the above-mentioned embodiment 1 or embodiment 2 are completed.

[0137] In summary, the present invention provides an adaptive data outlier detection method and terminal, which have the following beneficial effects:

[0138] 1. The threshold is now adjusted dynamically and does not rely on manual settings. As the system continues to run, the threshold will become more and more accurate;

[0139] 2. The system will select the appropriate model for detection based on the actual data situation, so that the system has better generalization ability and effectively improves the detection accuracy. The system adopts a modular design, allowing models to be easily added or replaced when needed;

[0140] 3. The calculation of the system is divided into real-time calculation and scheduled calculation. The real-time calculation part has a small amount of calculation, but it can filter out obvious abnormal data. Scheduled calculation will apply models to analyze a large amount of data, and can find complex anomalies. It can be performed when the system is idle and will not affect the normal operation of the system.

[0141] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An adaptive method for detecting outliers in data, It is characterized in that Includes steps: S1, receiving data in real time and screening according to the initial threshold range to obtain preliminary screening data; S2. Periodically pull historical data, and decompose the historical data based on a time series decomposition method to obtain the trend, seasonality, and residual of the historical data; S3. Determine a time series model to be used based on the trend, the seasonality and the residual; S4. Detect the initial screening data based on the time series model, determine abnormal values, and dynamically correct the initial threshold range according to the abnormal values.

2. The method for detecting an adaptive data outlier according to claim 1, It is characterized in that The step S2 specifically comprises: S21, regularly query the historical data list and pull the historical data; S22, based on the seasonal time series decomposition method, using the Loess smoothing algorithm to smooth the original time series of the historical data to obtain initial trend information; S23, subtracting the initial trend information from the original time series to obtain the detrended historical data; S24, using the Loess smoothing algorithm again to smooth the time series of the historical data after detrending to obtain initial seasonal information; S25, subtracting the initial seasonal information from the original time series to obtain the de-seasonalized historical data; S26, using the Loess smoothing algorithm again to smooth the time series of the deseasonalized historical data, and updating the initial trend information; S27, repeating steps S23-S26 until the trend information and the seasonal information converge to obtain final trend information and final seasonal information; S28, calculating the variance and residual of the final trend information and the variance and residual of the final seasonal information to obtain trend variance, trend residual, seasonal variance and seasonal residual; S29. Calculate the trend strength and seasonal strength of the historical data based on the trend variance, the trend residual, the seasonal variance and the seasonal residual.

3. The adaptive data outlier detection method according to claim 2, It is characterized in that The step S29 is specifically as follows: Calculating the ratio of the trend variance to the variance of the trend residual to obtain the trend strength of the historical data; The ratio of the seasonal variance to the variance of the seasonal residual is calculated to obtain the seasonal intensity of the historical data.

4. The method for detecting an adaptive data outlier according to claim 3, It is characterized in that The time series model includes a seasonal prediction model, a trend prediction model and a smoothing prediction model; The step S3 is specifically as follows: S31, if the seasonal intensity is greater than a first threshold, select the seasonal prediction model, otherwise proceed to step S32; S32. If the trend strength is greater than the first threshold, select the trend prediction model; otherwise, select the smooth prediction model.

5. The method for detecting an adaptive data outlier according to claim 4, It is characterized in that The historical data pulled at each time in step S2 is the historical data within a preset recent period; The seasonal prediction model includes a seasonal multi-source regression model, a seasonal autoregression model and a time series decomposition model; The trend prediction model includes a linear trend prediction model, a nonlinear trend prediction model and an autoregressive prediction model; The smoothing prediction model includes simple smoothing method, moving smoothing method and exponential smoothing method.

6. An adaptive data outlier detection terminal, It is characterized in that The invention comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented: S1, receiving data in real time and screening according to the initial threshold range to obtain preliminary screening data; S2. Periodically pull historical data, and decompose the historical data based on a time series decomposition method to obtain the trend, seasonality, and residual of the historical data; S3. Determine a time series model to be used based on the trend, the seasonality and the residual; S4. Detect the initial screening data based on the time series model, determine abnormal values, and dynamically correct the initial threshold range according to the abnormal values.

7. The adaptive data outlier detection terminal according to claim 6, It is characterized in that The step S2 specifically comprises: S21, regularly query the historical data list and pull the historical data; S22, based on the seasonal time series decomposition method, using the Loess smoothing algorithm to smooth the original time series of the historical data to obtain initial trend information; S23, subtracting the initial trend information from the original time series to obtain the detrended historical data; S24, using the Loess smoothing algorithm again to smooth the time series of the historical data after detrending to obtain initial seasonal information; S25, subtracting the initial seasonal information from the original time series to obtain the de-seasonalized historical data; S26, using the Loess smoothing algorithm again to smooth the time series of the deseasonalized historical data, and updating the initial trend information; S27, repeating steps S23-S26 until the trend information and the seasonal information converge to obtain final trend information and final seasonal information; S28, calculating the variance and residual of the final trend information and the variance and residual of the final seasonal information to obtain trend variance, trend residual, seasonal variance and seasonal residual; S29. Calculate the trend strength and seasonal strength of the historical data based on the trend variance, the trend residual, the seasonal variance and the seasonal residual.

8. The adaptive data outlier detection terminal according to claim 7, It is characterized in that The step S29 is specifically as follows: Calculating the ratio of the trend variance to the variance of the trend residual to obtain the trend strength of the historical data; The ratio of the seasonal variance to the variance of the seasonal residual is calculated to obtain the seasonal intensity of the historical data.

9. The adaptive data outlier detection terminal according to claim 8, It is characterized in that The time series model includes a seasonal prediction model, a trend prediction model and a smoothing prediction model; The step S3 is specifically as follows: S31, if the seasonal intensity is greater than a first threshold, select the seasonal prediction model, otherwise proceed to step S32; S32. If the trend strength is greater than the first threshold, select the trend prediction model; otherwise, select the smooth prediction model.

10. The adaptive data outlier detection terminal according to claim 9, It is characterized in that The historical data pulled at each time in step S2 is the historical data within a preset recent period; The seasonal prediction model includes a seasonal multi-source regression model, a seasonal autoregression model and a time series decomposition model; The trend prediction model includes a linear trend prediction model, a nonlinear trend prediction model and an autoregressive prediction model; The smoothing prediction model includes simple smoothing method, moving smoothing method and exponential smoothing method.

Citation Information

Cited By

  • Equipment abnormity intelligent detection method based on periodic data

    CN120045942A

  • Matching threshold value adjusting method and system for post portrait dynamic updating

    CN120146815A