Data anomaly detection method and device, equipment, storage medium and program product

Abstract

The invention discloses a data anomaly detection method, device and equipment, a storage medium and a program product, anomaly detection is performed on to-be-evaluated data by using multiple models, multiple data anomaly detection technologies are combined, statistical characteristics, long-term trend, seasonal change and accumulative change are comprehensively considered, and the data anomaly detection accuracy is improved. The comprehensiveness, accuracy and robustness of anomaly detection are improved, and the problem that a single method cannot cope with a complex data mode is effectively solved. Besides, based on historical data, a weight is distributed for each evaluation model according to the importance or performance, and a target evaluation result is selected from a plurality of candidate evaluation results through a weighted voting mechanism, so that false alarm and missing report can be effectively reduced, the accuracy and reliability of anomaly detection are improved, the dependence on a single method or threshold is reduced, and the accuracy and reliability of anomaly detection are improved. And stronger real-time adaptive capacity is provided under the condition that the data flow dynamically changes.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method, apparatus, device, storage medium, and program product for detecting data anomalies. Background Technology

[0002] With the widespread application of artificial intelligence (AI) technology, time-series data is increasingly used across various industries. Time-series data processing is a crucial application area of ​​AI. For example, by monitoring the business data continuously generated during the operation of a business system, anomalies can be detected and addressed promptly, ensuring the stability of the system. Current technologies for identifying anomalies in time-series data typically follow these steps: calculating daily time-series values ​​and using the historical average as the periodic average; then calculating the difference between the current data and the periodic average; and manually setting a threshold based on business experience. If the difference exceeds this threshold, the observation is marked as an anomaly. While existing methods can identify anomalous data, the calculation of the periodic average may be affected by data noise, leading to inaccurate anomaly detection and an inability to effectively distinguish between noise and actual anomalies, thus increasing the possibility of false positives and false negatives. Summary of the Invention

[0003] The purpose of this invention is to provide a data anomaly detection method, apparatus, device, storage medium, and program product that can effectively reduce false alarms and false negatives and improve the accuracy of data anomaly detection.

[0004] To achieve the above objectives, embodiments of the present invention provide a data anomaly detection method, comprising:

[0005] Obtain the data to be evaluated and historical data;

[0006] The data to be evaluated is input into at least two evaluation models to obtain at least two candidate evaluation results;

[0007] The confidence level is obtained based on the number of correct detections of each evaluation model on historical data, and the model weight of the evaluation model is determined using the confidence level.

[0008] Based on the model weights, a target evaluation result is selected from at least two candidate evaluation results, and the target evaluation result is used as the detection result of the data to be evaluated.

[0009] As an improvement to the above scheme, the at least two evaluation models are any two or three of the following: moving average model, time series forecasting model, and cumulative monitoring model.

[0010] As an improvement to the above scheme, when at least two evaluation models include the moving average model, the step of generating the corresponding candidate evaluation result using the moving average model includes:

[0011] Calculate the moving average and standard deviation of the historical data;

[0012] The evaluation range is determined based on the moving average and the standard deviation;

[0013] The data to be evaluated is compared with the evaluation range to generate corresponding candidate evaluation results.

[0014] As an improvement to the above scheme, the evaluation range consists of a first upper limit and a first lower limit, and determining the evaluation range based on the moving average and the standard deviation includes:

[0015] The first upper limit value is determined based on the moving average, the standard deviation, and a preset first fixed value;

[0016] The first lower limit value is determined based on the moving average, the standard deviation, and a preset second fixed value.

[0017] As an improvement to the above scheme, when at least two evaluation models include the time series prediction model, the step of generating corresponding candidate evaluation results through the time series prediction model includes:

[0018] The historical data is input into the time series prediction model so that the time series prediction model outputs predicted values;

[0019] Calculate the error between the predicted value and the data to be evaluated;

[0020] The error value is compared with a preset error threshold to generate a corresponding candidate evaluation result.

[0021] As an improvement to the above scheme, when at least two evaluation models include the cumulative sum monitoring model, the step of generating corresponding candidate evaluation results through the cumulative sum monitoring model includes:

[0022] The baseline value and deviation range are determined based on the historical data.

[0023] Calculate the deviation between the data to be evaluated and the benchmark value;

[0024] The deviation values ​​are added to the reference cumulative sum to obtain the corresponding target cumulative sum; wherein, the reference cumulative sum is the deviation value calculated when evaluating the previous data to be evaluated;

[0025] The target cumulative sum is compared with the deviation range to generate the corresponding candidate evaluation results.

[0026] As an improvement to the above scheme, the reference cumulative sum includes a reference positive cumulative sum and a reference negative cumulative sum, the target cumulative sum includes a target positive cumulative sum and a target negative cumulative sum, and the deviation range consists of a second upper limit value and a second lower limit value; the step of comparing the target cumulative sum with the deviation range to generate corresponding candidate evaluation results includes:

[0027] The target positive cumulative sum is compared with the second upper limit value, and the target negative cumulative sum is compared with the second lower limit value to generate corresponding candidate evaluation results.

[0028] As an improvement to the above scheme, the step of obtaining the corresponding confidence level based on the number of correct detections of historical data by each evaluation model includes:

[0029] Obtain the number of correct detections for each evaluation model on historical data;

[0030] The confidence level of the current evaluation model is determined by the ratio of the number of correct detections to the total number of detections.

[0031] As an improvement to the above solution, the method further includes:

[0032] Calculate the confidence difference between the current confidence level and the initial confidence level;

[0033] The model weights are updated using the confidence difference.

[0034] To achieve the above objectives, embodiments of the present invention also provide a data anomaly detection device, comprising:

[0035] The data acquisition module is used to acquire the data to be evaluated and historical data;

[0036] The candidate evaluation result generation module is used to input the data to be evaluated into at least two evaluation models to obtain at least two candidate evaluation results;

[0037] The model weight determination module is used to obtain the corresponding confidence level based on the number of correct detections of each evaluation model on historical data, and to determine the model weight of the evaluation model using the confidence level.

[0038] The detection result generation module is used to select a target evaluation result from at least two candidate evaluation results according to the model weights, and use the target evaluation result as the detection result of the data to be evaluated.

[0039] To achieve the above objectives, embodiments of the present invention also provide a data anomaly detection device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the data anomaly detection method as described in any of the above embodiments.

[0040] To achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the data anomaly detection method as described in any of the above embodiments.

[0041] To achieve the above objectives, embodiments of the present invention also provide a computer program product, including computer instructions, which, when executed by a processor, implement the data anomaly detection method as described in any of the above embodiments.

[0042] Compared to existing technologies, the data anomaly detection method, apparatus, device, storage medium, and program products disclosed in this invention utilize multiple models to detect anomalies in the data to be evaluated. By combining various data anomaly detection techniques and comprehensively considering statistical characteristics, long-term trends, seasonal variations, and cumulative changes, the comprehensiveness, accuracy, and robustness of anomaly detection are improved, effectively solving the problem that a single method cannot handle complex data patterns. Furthermore, by assigning a weight to each evaluation model based on its importance or performance using historical data, and selecting the target evaluation result from multiple candidate evaluation results through a weighted voting mechanism, false positives and false negatives can be effectively reduced, improving the accuracy and reliability of anomaly detection, reducing reliance on a single method or threshold, and providing stronger real-time adaptability in the face of dynamically changing data streams. Attached Figure Description

[0043] Figure 1 This is a flowchart of a data anomaly detection method provided in an embodiment of the present invention;

[0044] Figure 2 This is another flowchart of a data anomaly detection method provided in an embodiment of the present invention;

[0045] Figure 3 This is a flowchart of data evaluation using a moving average model provided in an embodiment of the present invention;

[0046] Figure 4 This is a flowchart of evaluating data using a time-series prediction model, provided in an embodiment of the present invention;

[0047] Figure 5 This is a flowchart of the data evaluation using a cumulative monitoring model provided in an embodiment of the present invention;

[0048] Figure 6 This is a structural block diagram of a data anomaly detection device provided in an embodiment of the present invention;

[0049] Figure 7 This is a structural block diagram of a data anomaly detection device provided in an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] See Figure 1 , Figure 1 This is a flowchart of a data anomaly detection method provided in an embodiment of the present invention. The data anomaly detection method includes:

[0052] S1. Obtain the data to be evaluated and historical data;

[0053] S2. Input the data to be evaluated into at least two evaluation models to obtain at least two candidate evaluation results;

[0054] S3. Obtain the corresponding confidence level based on the number of correct detections of each evaluation model on historical data, and use the confidence level to determine the model weight of the evaluation model;

[0055] S4. Based on the model weights, select a target evaluation result from at least two candidate evaluation results, and use the target evaluation result as the detection result of the data to be evaluated.

[0056] For example, the at least two evaluation models are any two or three of the following: a moving average model, a time series forecasting model, and a cumulative and monitoring model. See also Figure 2 , Figure 2 This is another flowchart of a data anomaly detection method provided in this embodiment of the invention. The data to be evaluated is time-series data. After obtaining the data to be evaluated, this data is also used as historical data storage. After adaptive parameter adjustment (or direct input without adaptive adjustment), the data to be evaluated is adapted to the input of different evaluation models. Then, the data to be evaluated is input into at least two evaluation models, such as... Figure 2The input is fed into three evaluation models: a moving average model, a time series prediction model, and a cumulative monitoring model. Then, the candidate evaluation results output by the three models are weighted and voted on. The candidate evaluation result with the highest number of votes and the highest weight is selected as the target evaluation result, and this target evaluation result is used as the detection result of the data to be evaluated.

[0057] In this embodiment of the invention, by integrating at least two evaluation models, noise filtering is applied to the time-series data to reduce invalid fluctuations. Then, a confidence-weighted mechanism and an adaptive weighting mechanism are used to dynamically adjust the weight configuration, and finally, an outlier detection results are output through a voting mechanism. This method overcomes the problems of traditional methods, such as sensitivity to data noise and reliance on manual experience for threshold setting. It can effectively reduce false positives and false negatives, improve detection accuracy, and provide stronger real-time adaptability in the face of dynamic changes in the data stream.

[0058] It is worth noting that the moving average model combines moving average and standard deviation algorithms. For data sources with high noise levels, the moving average window is widened, and the standard deviation threshold is increased to reduce false alarms caused by noise. The time series prediction model is the ARIMA algorithm model or other existing models capable of data prediction. This model increases tolerance for high-noise data, reduces sensitivity to small prediction deviations, and focuses on anomalies that significantly deviate from predicted values. The cumulative sum monitoring model is the CUSUM algorithm model or other existing models capable of cumulative sum monitoring. For frequently occurring but minimally sized noise, the control limit is increased to filter out small fluctuations and focus only on significant cumulative changes.

[0059] See Figure 3 , Figure 3 This is a flowchart of evaluating data using a moving average model provided in an embodiment of the present invention. When at least two evaluation models include the moving average model, the step of generating corresponding candidate evaluation results using the moving average model includes:

[0060] S11. Calculate the moving average and standard deviation of the historical data;

[0061] S12. Determine the evaluation range based on the moving average and the standard deviation;

[0062] S13. Compare the data to be evaluated with the evaluation range to generate corresponding candidate evaluation results.

[0063] For example, this process smooths the data and identifies significant deviations by using a sliding window of historical data to calculate the mean and dispersion of the data at the current point in time. Moving averages and standard deviations help reduce the impact of data noise and provide a stable benchmark for anomaly detection. For each point in time containing the data to be evaluated, the moving average and standard deviation of historical data from the previous period are calculated, and then the evaluation range is determined based on the moving average and standard deviation. If the data to be evaluated is not within the evaluation range, the candidate evaluation result is: the data to be evaluated is an outlier; if the data to be evaluated is within the evaluation range, the candidate evaluation result is: the data to be evaluated is a normal value.

[0064] Furthermore, the moving average SMA t The calculation process satisfies the following formula:

[0065]

[0066] Where, x t-i It is the value of the time series at time ti, where t is the current time point corresponding to the data to be evaluated, i is the time within the moving window, and n is the size of the moving window. The larger the moving window, the more historical data is captured, and vice versa.

[0067] Furthermore, the calculation of the standard deviation σ follows the formula:

[0068]

[0069] Where, x j Let μ represent the j-th data point in the historical data, where j = 1, 2, ..., m, and m is the total number of data points in the historical data; μ is the mean of the historical data.

[0070] Specifically, the evaluation range consists of a first upper limit and a first lower limit. Determining the evaluation range based on the moving average and the standard deviation includes: determining the first upper limit based on the moving average, the standard deviation, and a preset first fixed value; and determining the first lower limit based on the moving average, the standard deviation, and a preset second fixed value.

[0071] For example, the first fixed value and the second fixed value can be preset according to empirical values, and the first fixed value and the second fixed value are opposites of each other. For example, the first fixed value is k, and the second fixed value is -k, where k is a constant, usually 2 or 3. The evaluation range satisfies [T min ,T max ], then the first upper limit value T max and the first lower limit value T min The calculation process satisfies the following formula:

[0072] T max =(SMA) t +k)*σ (3);

[0073] T min =(SMA) t -k)*σ (4).

[0074] See Figure 4 , Figure 4 This is a flowchart of evaluating data using a time-series prediction model provided in an embodiment of the present invention. When at least two evaluation models include the time-series prediction model, the steps for generating corresponding candidate evaluation results using the time-series prediction model include:

[0075] S21. Input the historical data into the time series prediction model so that the time series prediction model outputs a predicted value;

[0076] S22. Calculate the error value between the predicted value and the data to be evaluated;

[0077] S23. Compare the error value with a preset error threshold to generate a corresponding candidate evaluation result.

[0078] For example, the ARIMA algorithm model is used as an example to illustrate the application of the ARIMA algorithm for modeling and predicting time series data. The ARIMA algorithm combines three parts: autoregression (AR), differencing (I), and moving average (MA), and can handle different types of time series data. An ARIMA model is often represented as ARIMA(p,d,q), where p is the order of the autoregressive part; d is the differencing order, used to station the time series; and q is the order of the moving average part. This algorithm generates predictions for future data by analyzing trends and seasonality in the data, thereby helping to identify significant deviations between predicted and actual values. This allows the system to capture the impact of long-term trends and periodic changes on anomaly detection. The specific steps are as follows:

[0079] ① Data preprocessing: Difference the time series data to ensure data stability;

[0080] ② Model identification and estimation: Select appropriate p, d, and q parameters to establish an ARIMA model;

[0081] ③ Model diagnosis: Check the model residuals to ensure they conform to white noise characteristics;

[0082] ④ Prediction and Detection: Prediction is performed using the ARIMA model, followed by the execution of steps S21 to S23 above. If the error value is greater than the error threshold, the candidate evaluation result is: the data to be evaluated is an outlier; if the error value is less than or equal to the error threshold, the candidate evaluation result is: the data to be evaluated is a normal value.

[0083] Furthermore, the autoregressive (AR) part uses past values ​​to predict future values. It satisfies the following formula:

[0084]

[0085] Among them, Y t c is the current value of the time series; c is a constant term. These are the autoregressive coefficients; p is the order of the autoregressive component; ∈ t It's white noise.

[0086] Furthermore, the differencing (I) part is used to station the time series data, eliminating trends and seasonality. It satisfies the following formula:

[0087]

[0088] in, d is the current value of the time series before and after the difference; d is the number of differences; when d=1, it means that one difference operation is performed, that is, the calculation of formula (6) is performed once; when d=2, it means that two differences are performed, and so on.

[0089] Furthermore, the moving average (MA) part: uses past forecast errors to predict future values. It satisfies the following formula:

[0090]

[0091] Where, θ i q is the moving average coefficient; q is the order of the moving average component.

[0092] See Figure 5 , Figure 5 This is a flowchart of an embodiment of the present invention for evaluating data using a cumulative sum monitoring model. When at least two evaluation models include the cumulative sum monitoring model, the step of generating corresponding candidate evaluation results through the cumulative sum monitoring model includes:

[0093] S31. Determine the baseline value and deviation range based on the historical data;

[0094] S32. Calculate the deviation between the data to be evaluated and the benchmark value;

[0095] S33. The deviation value is added to the reference cumulative sum to obtain the corresponding target cumulative sum; wherein, the reference cumulative sum is the deviation value calculated when evaluating the previous data to be evaluated;

[0096] S34. Compare the cumulative sum of the targets with the deviation range to generate corresponding candidate evaluation results.

[0097] For example, the CUSUM algorithm model is used as an example to illustrate the cumulative sum monitoring model. The CUSUM algorithm calculates the cumulative sum of deviations between observed values ​​and target values, and then uses the statistical properties of the cumulative sum to detect anomalies. This algorithm is particularly suitable for monitoring gradually changing anomalies and can effectively identify significant change points in the data. The benchmark value is the average of all or part of the historical data, or the mode of all or part of the historical data, i.e., the number that occurs most frequently. The deviation range can be determined based on the benchmark value, such as benchmark value ± A, where A is a constant, such as A taking the value 2 or 3.

[0098] Specifically, the reference cumulative sum includes a reference positive cumulative sum and a reference negative cumulative sum, the target cumulative sum includes a target positive cumulative sum and a target negative cumulative sum, and the deviation range is defined by a second upper limit value D. max Second lower limit value D min The composition; step S34 specifically includes: comparing the target positive cumulative sum with the second upper limit value, and comparing the target negative cumulative sum with the second lower limit value, to generate corresponding candidate evaluation results.

[0099] For example, the calculation process of the target positive cumulative sum and the target negative cumulative sum satisfies the following formula:

[0100]

[0101] in, Let the positive cumulative sum of the target at time t be ; Let be the reference positive cumulative sum at time t-1; The target negative cumulative sum at time t; x is the reference negative cumulative sum at time t-1; t t represents the data to be evaluated at time t; β represents the baseline value; f represents the offset, which is a fixed constant, usually set to half of the allowable deviation.

[0102] For example, step S34 includes the following cases:

[0103] 1) The target positive cumulative sum is greater than the second upper limit value, and the target negative cumulative sum is greater than the second lower limit value, that is, the following conditions are met: and The candidate evaluation result is: the data to be evaluated is an outlier;

[0104] 2) The target positive cumulative sum is greater than the second upper limit value, and the target negative cumulative sum is less than or equal to the second lower limit value, i.e., the following conditions are met: and The candidate evaluation result is: the data to be evaluated is an outlier;

[0105] 3) The target positive cumulative sum is less than or equal to the second upper limit value, and the target negative cumulative sum is greater than the second lower limit value, i.e., the following conditions are met: and The candidate evaluation result is: the data to be evaluated is an outlier;

[0106] 4) The target positive cumulative sum is less than or equal to the second upper limit value, and the target negative cumulative sum is less than or equal to the second lower limit value, i.e., the following conditions are met: and The candidate evaluation result is: the data to be evaluated is within the normal range.

[0107] Specifically, in step S3, obtaining the corresponding confidence level based on the number of correct detections of historical data by each evaluation model includes: obtaining the number of correct detections of historical data by each evaluation model; and using the ratio of the number of correct detections to the total number of detections as the confidence level of the current evaluation model.

[0108] For example, each algorithm assigns a confidence score to its detected candidate evaluation values; a higher score indicates greater confidence in the algorithm's detection of the outlier. In the initial detection phase, based on the algorithm's performance on historical data, an initial weight is assigned to each algorithm using a confidence-weighted mechanism. The confidence score is calculated according to the following formula:

[0109]

[0110] Among them, conf_score i Represents the confidence level of the i-th evaluation model; correct_detections i This represents the number of correct detections for the i-th evaluation model. Since the evaluation model outputs a candidate evaluation result each time, it will undergo subsequent evaluations to obtain the target evaluation result (i.e., step S4). If the candidate evaluation result matches the target evaluation result, the number of correct detections is incremented by 1. If the candidate evaluation result does not match the target evaluation result, the number of correct detections is incremented by 0. total_detections represents the total number of detections for the i-th evaluation model.

[0111] Specifically, determining the model weights of the evaluation model using the confidence level includes: assigning model weights to each evaluation model using the confidence level, satisfying the following formula:

[0112]

[0113] Among them, w i is the model weight of the i-th evaluation model; N is the total number of models.

[0114] Specifically, the method further includes: calculating the confidence difference between the current confidence level and the initial confidence level; and updating the model weights using the confidence difference.

[0115] For example, as time series data is updated, the detection environment and data characteristics constantly change. Static confidence weights alone cannot maintain efficiency indefinitely, necessitating the introduction of an adaptive voting weight mechanism. This adaptive voting mechanism fine-tunes the weights based on the detection results of each algorithm in the current environment: algorithms with better performance receive higher voting weights, while those with poorer performance have their weights reduced.

[0116]

[0117] in, This indicates that the updated model weights need to be applied to the next evaluation of the data to be evaluated. This represents the model weight of the i-th evaluation model at time t; η is the parameter value with the best performance, which is selected by traversing all possible values ​​using a grid search method, with the value range defined as (0, 1). This represents the confidence level of the i-th evaluation model at time t. This represents the initial confidence level of the i-th evaluation model.

[0118] Furthermore, after updating the model weights, to ensure that the sum of all weights is 1 and to guarantee the effectiveness of weighted voting, normalization is required, satisfying the following formula:

[0119]

[0120] in, This represents the model weights after normalization.

[0121] For example, suppose we use any two of the three evaluation models mentioned above, taking the moving average model and the time series prediction model as examples. After the data to be evaluated is input, the moving average model outputs a candidate evaluation result of "normal value," while the time series prediction model outputs a candidate evaluation result of "outlier value." Assuming the model weight of the moving average model is greater than that of the time series prediction model, then the candidate evaluation result output by the moving average model is taken as the target evaluation result, meaning the data to be evaluated is determined to be a normal value. Furthermore, the number of correct detections by the moving average model is incremented by 1.

[0122] For example, assuming that the above three evaluation models are used simultaneously, after the evaluation data is input, the candidate evaluation result output by the moving average model is "normal value", the candidate evaluation result output by the time series prediction model is "outlier value", and the candidate evaluation result output by the cumulative sum monitoring model is "outlier value". The following two situations exist:

[0123] 1) The model weights of the three models are not equal; assuming the model weights of the moving average model, time series prediction model, and cumulative sum monitoring model increase sequentially, then the candidate evaluation result output by the cumulative sum monitoring model is used as the target evaluation result, i.e., the data to be evaluated is determined to be an outlier. Furthermore, the number of correct detections by the time series prediction model and the number of correct detections by the cumulative sum monitoring model are both increased by 1.

[0124] 2) At least two models have equal model weights, and the model with the largest weight is considered the most significant. Assuming the cumulative sum monitoring model has the smallest weight, and the time series prediction model and the moving average model have equal weights, since the two larger models have equal weights and opposite candidate evaluation results, the candidate evaluation result output by the cumulative sum monitoring model is used as the target evaluation result, i.e., the data to be evaluated is determined to be an outlier. Furthermore, the number of correct detections for the time series prediction model and the cumulative sum monitoring model are incremented by 1.

[0125] Compared with existing technologies, the data anomaly detection method provided in this invention has the following advantages:

[0126] 1. Reduce reliance on manually set thresholds. Current methods require manual threshold setting based on business experience, which is not only time-consuming but also prone to deviations due to differences in individual experience. This invention reduces reliance on manual thresholds through an automated weighted voting mechanism, improving the objectivity and reliability of the results.

[0127] 2. This invention integrates multiple detection technologies, and different integrated algorithms have their advantages in different scenarios. For example, the CUSUM algorithm can capture small, persistent changes, moving averages and standard deviations are suitable for identifying abrupt changes, while the ARIMA algorithm can analyze and predict future trends in time series data. Using a weighted voting mechanism to integrate at least two of these algorithms to evaluate data can capture anomalous patterns that might be missed by a single model, making the overall detection strategy more comprehensive.

[0128] 3. This invention improves adaptability by dynamically adjusting weights. Traditional methods often rely on comparisons with fixed historical data, lacking adaptability. This invention sets up a time window and a dynamic weight adjustment mechanism, which can automatically adjust weights based on the algorithm's real-time performance and historical results. This allows it to adapt to changes in data patterns, continuously optimize performance, and maintain efficient operation in constantly changing data environments.

[0129] See Figure 6 , Figure 6 This is a structural block diagram of a data anomaly detection device 100 provided in an embodiment of the present invention. The data anomaly detection device 100 includes:

[0130] Data acquisition module 11 is used to acquire the data to be evaluated and historical data;

[0131] The candidate evaluation result generation module 12 is used to input the data to be evaluated into at least two evaluation models to obtain at least two candidate evaluation results;

[0132] The model weight determination module 13 is used to obtain the corresponding confidence level based on the number of correct detections of each evaluation model on historical data, and to determine the model weight of the evaluation model using the confidence level.

[0133] The detection result generation module 14 is used to select a target evaluation result from at least two candidate evaluation results according to the model weights, and use the target evaluation result as the detection result of the data to be evaluated.

[0134] Specifically, the at least two evaluation models are any two or three of the following: moving average model, time series forecasting model, and cumulative monitoring model.

[0135] Specifically, when at least two evaluation models include the moving average model, the candidate evaluation result generation module 12 is used to: calculate the moving average and standard deviation of the historical data; determine the evaluation range based on the moving average and standard deviation; and compare the data to be evaluated with the evaluation range to generate corresponding candidate evaluation results.

[0136] Specifically, the evaluation range consists of a first upper limit and a first lower limit. Determining the evaluation range based on the moving average and the standard deviation includes: determining the first upper limit based on the moving average, the standard deviation, and a preset first fixed value; and determining the first lower limit based on the moving average, the standard deviation, and a preset second fixed value.

[0137] Specifically, when at least two evaluation models include the time series prediction model, the candidate evaluation result generation module 12 is used to: input the historical data into the time series prediction model so that the time series prediction model outputs a predicted value; calculate the error value between the predicted value and the data to be evaluated; and compare the error value with a preset error threshold to generate a corresponding candidate evaluation result.

[0138] Specifically, when at least two evaluation models include the cumulative sum monitoring model, the candidate evaluation result generation module 12 is used to: determine a benchmark value and a deviation range based on the historical data; calculate the deviation value between the data to be evaluated and the benchmark value; add the deviation value to a reference cumulative sum to obtain a corresponding target cumulative sum; wherein the reference cumulative sum is the deviation value calculated when evaluating the previous data to be evaluated; and compare the target cumulative sum with the deviation range to generate a corresponding candidate evaluation result.

[0139] Specifically, the reference cumulative sum includes a reference positive cumulative sum and a reference negative cumulative sum, the target cumulative sum includes a target positive cumulative sum and a target negative cumulative sum, and the deviation range consists of a second upper limit value and a second lower limit value; comparing the target cumulative sum with the deviation range to generate a corresponding candidate evaluation result includes: comparing the target positive cumulative sum with the second upper limit value, and comparing the target negative cumulative sum with the second lower limit value to generate a corresponding candidate evaluation result.

[0140] Specifically, the model weight determination module 13 is used to: obtain the number of correct detections of each evaluation model on historical data; and use the ratio of the number of correct detections to the total number of detections as the confidence level of the current evaluation model.

[0141] Specifically, the data anomaly detection device 100 further includes:

[0142] The model weight update module, used in the method, further includes: calculating the confidence difference between the current confidence level and the initial confidence level; and updating the model weights using the confidence difference.

[0143] It is worth noting that the working process of each module in the data anomaly detection device 100 described in the embodiments of the present invention can refer to the working process of the data anomaly detection method described in the above embodiments, and will not be repeated here.

[0144] See Figure 7 , Figure 7 This is a structural block diagram of a data anomaly detection device 200 provided in an embodiment of the present invention. The data anomaly detection device 200 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described embodiments of the data anomaly detection methods, such as steps S1-S3, S11-S13, S21-S23, and S31-S34.

[0145] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the data anomaly detection device 200.

[0146] The data anomaly detection device 200 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the data anomaly detection device 200 and does not constitute a limitation on the data anomaly detection device 200. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the data anomaly detection device 200 may also include input / output devices, network access devices, buses, etc.

[0147] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the data anomaly detection device 200, connecting all parts of the data anomaly detection device 200 via various interfaces and lines.

[0148] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the data anomaly detection device 200 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0149] If the modules / units integrated in the data anomaly detection device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0150] This invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the data anomaly detection method as described in the above embodiments.

[0151] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for detecting data anomalies, characterized in that, include: Obtain the data to be evaluated and historical data; The data to be evaluated is input into at least two evaluation models to obtain at least two candidate evaluation results; The confidence level is obtained based on the number of correct detections of each evaluation model on historical data, and the model weight of the evaluation model is determined using the confidence level. Based on the model weights, a target evaluation result is selected from at least two candidate evaluation results, and the target evaluation result is used as the detection result of the data to be evaluated.

2. The data anomaly detection method as described in claim 1, characterized in that, The at least two evaluation models are any two or three of the following: moving average model, time series forecasting model, and cumulative monitoring model.

3. The data anomaly detection method as described in claim 2, characterized in that, When at least two evaluation models include the moving average model, the steps for generating corresponding candidate evaluation results using the moving average model include: Calculate the moving average and standard deviation of the historical data; The evaluation range is determined based on the moving average and the standard deviation; The data to be evaluated is compared with the evaluation range to generate corresponding candidate evaluation results.

4. The data anomaly detection method as described in claim 3, characterized in that, The evaluation range consists of a first upper limit and a first lower limit, and determining the evaluation range based on the moving average and the standard deviation includes: The first upper limit value is determined based on the moving average, the standard deviation, and a preset first fixed value; The first lower limit value is determined based on the moving average, the standard deviation, and a preset second fixed value.

5. The data anomaly detection method as described in claim 2, characterized in that, When at least two evaluation models include the time-series prediction model, the steps for generating corresponding candidate evaluation results using the time-series prediction model include: The historical data is input into the time series prediction model so that the time series prediction model outputs predicted values; Calculate the error between the predicted value and the data to be evaluated; The error value is compared with a preset error threshold to generate a corresponding candidate evaluation result.

6. The data anomaly detection method as described in claim 2, characterized in that, When at least two evaluation models include the cumulative sum monitoring model, the steps for generating corresponding candidate evaluation results using the cumulative sum monitoring model include: The baseline value and deviation range are determined based on the historical data. Calculate the deviation between the data to be evaluated and the benchmark value; The deviation values ​​are added to the reference cumulative sum to obtain the corresponding target cumulative sum; wherein, the reference cumulative sum is the deviation value calculated when evaluating the previous data to be evaluated; The target cumulative sum is compared with the deviation range to generate the corresponding candidate evaluation results.

7. The data anomaly detection method as described in claim 6, characterized in that, The reference cumulative sum includes a reference positive cumulative sum and a reference negative cumulative sum, the target cumulative sum includes a target positive cumulative sum and a target negative cumulative sum, and the deviation range consists of a second upper limit value and a second lower limit value; comparing the target cumulative sum with the deviation range to generate corresponding candidate evaluation results includes: The target positive cumulative sum is compared with the second upper limit value, and the target negative cumulative sum is compared with the second lower limit value to generate corresponding candidate evaluation results.

8. The data anomaly detection method as described in claim 1, characterized in that, The process of obtaining the corresponding confidence level based on the number of correct detections of historical data by each evaluation model includes: Obtain the number of correct detections for each evaluation model on historical data; The confidence level of the current evaluation model is determined by the ratio of the number of correct detections to the total number of detections.

9. The data anomaly detection method as described in claim 8, characterized in that, The method further includes: Calculate the confidence difference between the current confidence level and the initial confidence level; The model weights are updated using the confidence difference.

10. A data anomaly detection device, characterized in that, include: The data acquisition module is used to acquire the data to be evaluated and historical data; The candidate evaluation result generation module is used to input the data to be evaluated into at least two evaluation models to obtain at least two candidate evaluation results; The model weight determination module is used to obtain the corresponding confidence level based on the number of correct detections of each evaluation model on historical data, and to determine the model weight of the evaluation model using the confidence level. The detection result generation module is used to select a target evaluation result from at least two candidate evaluation results according to the model weights, and use the target evaluation result as the detection result of the data to be evaluated.

11. A data anomaly detection device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the data anomaly detection method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the data anomaly detection method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the data anomaly detection method as described in any one of claims 1 to 9.

Images