Intelligent anomaly detection system and method for time series data based on large model
Through the intelligent timing data abnormality detection system based on large models, identify data characteristics and adjust detection algorithms, the problem that traditional monitoring systems cannot adapt to data changes is solved, and high accuracy and flexibility abnormality detection is achieved, reducing operational costs and enhancing The reliability of business processes.
Patent Information
- Application Number
- CN202411019014.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-26
AI Technical Summary
Traditional monitoring systems rely on preset thresholds for abnormal detection and are unable to adapt to dynamic changes in data, resulting in false alarms and missed alarms, affecting the effectiveness and timeliness of decision-making.
Using a large model-based intelligent anomaly detection system for time series data, the periodicity, driftness and stationarity of data are identified through the feature analysis module, intelligently select and adjust the anomaly detection algorithm, and dynamic loading and parallel training are used for algorithm factory mode, using incremental Learning and parameter optimization methods enable the model to quickly adapt to data changes.
It improves the accuracy and flexibility of abnormal detection, realizes real-time detection and early warning, reduces operational costs, and enhances the efficiency and reliability of business processes.
Smart Images

Figure CN119026045A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data anomaly detection, and in particular to a large model-based time series data intelligent anomaly detection system and method thereof. Background Art
[0002] In modern enterprise operations, business time series data, including user behavior, transaction records, system logs, etc., plays a vital role. This type of data not only reflects the real-time status of business activities, but also contains rich information that can be used to predict trends, optimize processes, and improve user experience. However, with the continuous development of business and the rapid growth of data volume, ensuring the health and stability of these data streams has become a major challenge.
[0003] Traditional monitoring systems mostly rely on preset thresholds to trigger alarms. This method is simple and direct but has obvious limitations. Fixed thresholds often cannot adapt to dynamic changes in data, especially when facing seasonal fluctuations, emergencies or business model adjustments, which can easily generate a large number of false positives or false negatives, thus affecting the effectiveness and timeliness of decision-making.
[0004] In recent years, the rapid development of artificial intelligence (AI) and machine learning (ML) technologies has provided new solutions for anomaly detection in business time series data. AI algorithms can learn complex patterns and rules from historical data and identify abnormal behaviors outside of regular patterns without manually setting fixed thresholds. This approach can not only improve the accuracy and timeliness of anomaly detection, but also automatically adjust when data features change to adapt to the evolving business environment. Summary of the invention
[0005] In order to solve the above-mentioned problems existing in traditional monitoring systems, the present invention provides a time series data intelligent anomaly detection system and method based on a large model. The core goal of the system is to improve the accuracy and timeliness of anomaly detection in a dynamically changing business environment, and it is particularly suitable for business scenarios that require high reliability and response speed.
[0006] To achieve the above object, the present invention adopts the following technical solutions:
[0007] In one embodiment of the present invention, a time series data intelligent anomaly detection system based on a large model is proposed, and the system includes:
[0008] The feature analysis module is used to identify the periodicity of time series data using the moving average method, detect the drift of time series data using the median filter method, and determine the stability of time series data using the ADF test;
[0009] The algorithm selection module is used to select the absolute median difference, box plot or extreme value theory according to the skewness value of the data distribution, and adjust the parameters of the selected anomaly detection algorithm according to the analysis results of periodicity, drift and stationarity;
[0010] Model training module, which uses the algorithm factory mode to support dynamic loading and parallel training of different anomaly detection algorithms; for periodic data, segmentation and aggregation processing are used to enhance model training; for drifting data, incremental learning methods are used to gradually adapt the model to data changes; parameter optimization methods are used to ensure optimal model performance;
[0011] The real-time detection module is used to continuously receive real-time time series data, load the trained model after preprocessing for anomaly detection, and output the results immediately once an anomaly is detected.
[0012] Furthermore, the following steps are used to identify the periodicity of time series data:
[0013] Use the moving average method to extract trend components from time series data;
[0014] Subtract the original sequence of time series data from the trend component to obtain the residual sequence;
[0015] Perform cyclic shift on the residual sequence and perform vector point multiplication with the original sequence to calculate the autocorrelation sequence;
[0016] Extract the local maximum peak of the autocorrelation sequence, and take the horizontal axis interval as the period T;
[0017] If the autocorrelation value at period T is greater than a given threshold, significant periodicity is considered to exist.
[0018] Furthermore, the following steps are used to detect the drift of time series data:
[0019] Select an appropriate window size, extract the median within the window and perform smoothing;
[0020] Check whether the smoothed series shows a strictly increasing or decreasing trend;
[0021] Check whether there is any point where the maximum value of the sequence on the left is less than the minimum value of the sequence on the right; if so, it is considered that the data has experienced a sudden increase drift; check whether there is any point where the minimum value of the sequence on the left is greater than the maximum value of the sequence on the right; if so, it is considered that the data has experienced a sudden decrease drift.
[0022] Furthermore, the following steps are used to select and adjust the appropriate anomaly detection algorithm:
[0023] (1) Calculate the skewness value of the data distribution and preliminarily select the algorithm based on the skewness value:
[0024] |Skewness|<0.5: use absolute median difference; 0.5≤|Skewness|<1: use box plot; |Skewness|≥1: use extreme value theory;
[0025] (2) According to the analysis results of periodicity, drift and stability, the preliminary algorithm is adjusted:
[0026] If cycles are evident: for absolute median difference and box plots, consider using a cyclical variant; for extreme value theory, consider performing a seasonal decomposition before applying extreme value theory;
[0027] If drift exists: For all algorithms, consider introducing an adaptive threshold mechanism; if drift is significant, online learning algorithms or change point detection and segmentation modeling may be prioritized;
[0028] If non-stationary: For absolute median difference and box plots, consider differencing or detrending before applying the algorithm; you may be tempted to choose an algorithm such as ARIMA or LSTM that can handle non-stationary data.
[0029] Furthermore, the following technical architecture is used to implement model training:
[0030] Adopting the algorithm factory mode, it supports dynamic loading of different anomaly detection algorithms; implements standardized training and detection interfaces for each algorithm; and supports parallel training of algorithms;
[0031] Design cycle detection function to automatically identify periodic patterns in data; support multiple methods for processing periodic data; implement segmentation and aggregation of data for detected cycles;
[0032] Design drift detection function to identify changes in data distribution; support dynamic adjustment of training window size; implement incremental learning method to make the model gradually adapt to data changes;
[0033] Implement multiple parameter optimization methods; support cross-validation to ensure the stability of parameter selection; and implement early stopping mechanism;
[0034] Design model serialization and deserialization mechanisms to support saving trained models to storage systems; implement version control to allow saving and loading of different versions of models; and manage model metadata.
[0035] In one embodiment of the present invention, a method for intelligent anomaly detection of time series data based on a large model is also proposed, and the method includes:
[0036] Use the moving average method to identify the periodicity of time series data, use the median filter method to detect the drift of time series data, and use the ADF test to determine the stationarity of time series data;
[0037] Select the absolute median difference, box plot or extreme value theory according to the skewness value of the data distribution, and adjust the parameters of the selected anomaly detection algorithm based on the analysis results of periodicity, drift and stationarity;
[0038] The algorithm factory mode is adopted to support dynamic loading and parallel training of different anomaly detection algorithms. For periodic data, segmentation and aggregation processing are used to enhance model training. For drifting data, incremental learning methods are used to gradually adapt the model to data changes. Parameter optimization methods are used to ensure that the model performance is optimal.
[0039] Continuously receive real-time time series data, load the trained model after preprocessing for anomaly detection, and output the results immediately once an anomaly is detected.
[0040] Furthermore, the following steps are used to identify the periodicity of time series data:
[0041] Use the moving average method to extract trend components from time series data;
[0042] Subtract the original sequence of time series data from the trend component to obtain the residual sequence;
[0043] Perform cyclic shift on the residual sequence and perform vector point multiplication with the original sequence to calculate the autocorrelation sequence;
[0044] Extract the local maximum peak of the autocorrelation sequence, and take the horizontal axis interval as the period T;
[0045] If the autocorrelation value at period T is greater than a given threshold, significant periodicity is considered to exist.
[0046] Furthermore, the following steps are used to detect the drift of time series data:
[0047] Select an appropriate window size, extract the median within the window and perform smoothing;
[0048] Check whether the smoothed series shows a strictly increasing or decreasing trend;
[0049] Check whether there is any point where the maximum value of the sequence on the left is less than the minimum value of the sequence on the right; if so, it is considered that the data has experienced a sudden increase drift; check whether there is any point where the minimum value of the sequence on the left is greater than the maximum value of the sequence on the right; if so, it is considered that the data has experienced a sudden decrease drift.
[0050] Furthermore, the following steps are used to select and adjust the appropriate anomaly detection algorithm:
[0051] (1) Calculate the skewness value of the data distribution and preliminarily select the algorithm based on the skewness value:
[0052] |Skewness|<0.5: use absolute median difference; 0.5≤|Skewness|<1: use box plot; |Skewness|≥1: use extreme value theory;
[0053] (2) According to the analysis results of periodicity, drift and stability, the preliminary algorithm is adjusted:
[0054] If cycles are evident: for absolute median difference and box plots, consider using a cyclical variant; for extreme value theory, consider performing a seasonal decomposition before applying extreme value theory;
[0055] If drift exists: For all algorithms, consider introducing an adaptive threshold mechanism; if drift is significant, online learning algorithms or change point detection and segmentation modeling may be prioritized;
[0056] If non-stationary: For absolute median difference and box plots, consider differencing or detrending before applying the algorithm; you may be tempted to choose an algorithm such as ARIMA or LSTM that can handle non-stationary data.
[0057] Furthermore, the following technical architecture is used to implement model training:
[0058] Adopting the algorithm factory mode, it supports dynamic loading of different anomaly detection algorithms; implements standardized training and detection interfaces for each algorithm; and supports parallel training of algorithms;
[0059] Design cycle detection function to automatically identify periodic patterns in data; support multiple methods for processing periodic data; implement segmentation and aggregation of data for detected cycles;
[0060] Design drift detection function to identify changes in data distribution; support dynamic adjustment of training window size; implement incremental learning method to make the model gradually adapt to data changes;
[0061] Implement multiple parameter optimization methods; support cross-validation to ensure the stability of parameter selection; and implement early stopping mechanism;
[0062] Design model serialization and deserialization mechanisms to support saving trained models to storage systems; implement version control to allow saving and loading of different versions of models; and manage model metadata.
[0063] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned large model-based intelligent anomaly detection method for time series data is implemented.
[0064] In one embodiment of the present invention, a computer-readable storage medium is also proposed, which stores a computer program for executing a large model-based intelligent anomaly detection method for time series data.
[0065] Beneficial effects:
[0066] 1. The present invention automatically identifies the periodicity, drift and stability characteristics of data, intelligently selects and adjusts the anomaly detection algorithm, ensures that the model can capture the inherent laws of the data, and improves the accuracy and flexibility of detection.
[0067] 2. The present invention supports dynamic loading and parallel training of algorithms, and adopts segmented aggregation, incremental learning and parameter optimization methods to ensure that the model quickly adapts to data changes, maintains high accuracy and reduces maintenance costs.
[0068] 3. The real-time detection module of the present invention continuously monitors data, loads the trained model for anomaly detection, responds immediately and outputs results, realizes early warning of abnormal events, and speeds up decision-making.
[0069] 4. Compared with traditional threshold monitoring, the present invention has higher anomaly detection accuracy and recall rate, can perform automatic inspections around the clock, adapt to various business scenarios, and provide strong support for business stability and sustainable development.
[0070] 5. The present invention reduces the need for manual monitoring and reduces operating costs through automated and intelligent anomaly detection, while improving the efficiency and reliability of business processes, especially in areas such as financial transactions, network security, and production control that require high reliability and response speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] Figure 1 It is an architecture diagram of the time series data intelligent anomaly detection system based on a large model of the present invention;
[0072] Figure 2 It is a flow chart of the intelligent anomaly detection method for time series data based on a large model of the present invention;
[0073] Figure 3 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION
[0074] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.
[0075] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.
[0076] According to the implementation of the present invention, a large-model-based intelligent anomaly detection system for time series data is proposed. Its core goal is to improve the accuracy and timeliness of anomaly detection in a dynamically changing business environment, especially for business scenarios that require high reliability and response speed. The design concept of the system is to automatically identify and adapt to the inherent laws of data through intelligent means to achieve early warning of anomalies, thereby providing instant decision support for the business team and ensuring the stable operation of the business.
[0077] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.
[0078] Figure 1 This is the architecture diagram of the time series data intelligent anomaly detection system based on a large model of the present invention. Figure 1 As shown, the system includes the following key modules:
[0079] 1. Feature analysis module: This module focuses on analyzing the periodicity, drift and stability characteristics of business time series data, providing a scientific basis for subsequent algorithm selection and model training. The moving average method is used to identify the periodicity of the data, the median filter method is used to detect the drift changes of the data, and the Augmented Dickey-Fuller (ADF) test is used to determine the stability of the data, ensuring a comprehensive grasp of the data characteristics.
[0080] Business time series data includes user behavior data, transaction records, system logs, and any data that reflects the real-time status of business activities, such as the number of user logins per day, transaction amounts, and system response time.
[0081] 2. Algorithm selection module: Based on the results of feature analysis, the system intelligently selects the most suitable anomaly detection algorithm. This includes selecting the median absolute difference (MAD), box plot or extreme value theory (EVT) according to the skewness value of the data distribution, and adjusting the algorithm parameters according to the feature analysis results of periodicity, drift and stationarity, such as introducing periodic variants, adaptive threshold mechanisms or selecting advanced algorithms that can handle non-stationary data such as ARIMA or LSTM.
[0082] The skewness value of data distribution can be calculated by statistical analysis. Skewness is a statistic that describes the symmetry of data distribution. It measures the degree of asymmetry of data distribution relative to normal distribution. In normal distribution, the skewness value is 0, indicating that the data distribution is symmetrical. When the skewness value is greater than 0, the data distribution is positively skewed, that is, the long tail is distributed on the right; when the skewness value is less than 0, the data distribution is negatively skewed, that is, the long tail is distributed on the left. In practice, many statistical software and programming languages such as R, Python (using scipy.stats.skew function), Excel (using SKEW function), etc. provide tools for directly calculating skewness, which can be directly applied to the data set to obtain the skewness value.
[0083] The choice and tuning of parameters depends on the chosen anomaly detection algorithm. For example:
[0084] For the median absolute difference (MAD), parameters may include multiples of the median to define the boundaries of the anomaly.
[0085] Boxplot may involve the upper and lower limits of the quartiles, as well as multiples of the IQR (interquartile range).
[0086] Extreme Value Theory (EVT), which may involve the estimation of tail exponents.
[0087] For ARIMA or LSTM models, parameters may include model order, number of hidden layer units, learning rate, etc.
[0088] 3. Model training module: The algorithm factory mode is adopted to support dynamic loading and parallel training of different anomaly detection algorithms to optimize model training efficiency. For periodic data, the system enhances model training through segmentation and aggregation processing; for data drift, an incremental learning strategy is adopted to allow the model to gradually adapt to data changes; and parameter optimization methods such as grid search and random search are used to ensure that the model performance is optimal. For stationary data, since the data is stationary, the entire data set can be used directly for model training without additional preprocessing steps.
[0089] 4. Real-time detection module: Continuously receive real-time business time series data, load the trained model after preprocessing for anomaly detection. The system regularly updates the model to adapt to slight drifts in the data to ensure the continued accuracy of the model. Once an anomaly is detected, the system immediately outputs the result so that the business team can respond quickly.
[0090] Through intelligent analysis and dynamic adjustment, this system can issue warnings at the early stages of business anomalies, providing the business team with immediate decision-making basis. Compared with traditional fixed threshold monitoring, this system has higher anomaly detection accuracy and recall rate, can achieve all-weather automatic inspections, and can flexibly adapt to various business scenarios, providing strong support for business stability and sustainable development. By timely detecting anomalies before problems worsen, it provides more valuable early warning signals for the business team, ensuring the smooth operation and continuous optimization of the business.
[0091] It should be noted that the implementation of the system can refer to the implementation of the following method, and the repeated parts will not be repeated. The term "module" used above can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the above embodiment is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceivable.
[0092] In order to explain the above-mentioned large model-based time series data intelligent anomaly detection system more clearly, a specific embodiment is used for illustration below. However, it is worth noting that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.
[0093] Example:
[0094] 1. Feature Analysis Module
[0095] This module is used to analyze the periodicity, drift and stability characteristics of time series data, providing a basis for subsequent algorithm selection and model training. Specifically, it includes:
[0096] 1. Periodic change analysis method
[0097] The present invention adopts the following steps to analyze the periodicity of time series data:
[0098] (1) Extract trend components: Use the moving average method to extract long-term trend items. For example, for the time series of the number of daily user logins, take the 7-day moving average as the trend component. Suppose we have the following set of data, representing the number of daily user logins for a website over 28 consecutive days:
[0099] Day: 1 2 3 4 5 6 7 8 910111213141516171819202122232425262728
[0100] Visits: 100120130140150160170100120130140150160170100120130140150160170100120130140150160170
[0101] Use the 7-day moving average method to extract the trend component. For example, for the 7th day, the trend component is the average of the number of user logins in the previous 7 days. Result example: Day: 7 8 91011121314
[0102] Trend: 138.57140141.43142.86144.29145.71147.14148.57
[0103] (2) Separate the residual sequence: Differ the original sequence from the trend component to obtain the residual sequence. For example:
[0104] Day: 7 8 9 10 11 12 13 14
[0105] Residual:31.43-40-21.43-12.86-14.29 4.29 12.86 21.43
[0106] (3) Calculate cyclic autocorrelation: Circularly shift the residual sequence and perform vector dot multiplication with the original sequence to calculate the autocorrelation sequence. This will help us identify whether there are periodic patterns in the data. For example:
[0107] Lag: 0 1 2 3 4 5 6 7 8 91011 121314ACF: 1.00.10.20.30.20.10.00.90.10.20.30.20.10.00.8
[0108] (4) Determine the period T: Extract the local maximum peak of the autocorrelation sequence and take the horizontal axis interval as the period. In this example, we found that the autocorrelation value at the lag of 7 days (Lag: 7) is 0.9, while the autocorrelation value at the lag of 14 days (Lag: 14) is 0.8. The autocorrelation values at 7 days and 14 days are significantly higher than 0.5, which confirms the 7-day periodicity.
[0109] (5) Verify periodicity: If the autocorrelation value at period T is greater than a given threshold (such as 0.5), it is considered that there is significant periodicity. In this example, we found that the autocorrelation values at 7 days and 14 days are significantly higher than 0.5, which confirms the 7-day periodicity.
[0110] 2. Drift change detection method
[0111] This paper uses a median filter-based technique to detect drift in time series data, aiming to capture unexpected data changes that occur over time. Such changes may be signals that indicate degradation of system performance or other types of abnormal conditions. With this method, we can promptly discover and analyze potential problem trends in time series data.
[0112] (1) Median smoothing: Select an appropriate window size (such as 30 days) and extract the median within the window. Suppose we are monitoring the number of user logins, and the data is as follows: Day: 1234567891011 1213 1415 1617181920 Time (ms): 100105110115120125130135140145150155160165170175180185190195
[0113] We use a 30-day window for median smoothing. On the 30th day, we calculate the median of the number of logins over the previous 30 days. After that, every day we roll the window forward by one day and recalculate the median.
[0114] (2) Determine the trend of the smoothed sequence: Check whether the smoothed sequence shows a strictly increasing or decreasing trend. This helps determine whether the data is undergoing a sustained change. For example, continuing to use the above data, assume that after smoothing the 30-day median, we get the following sequence:
[0115] Smoothed Day:303132...59
[0116] Smoothed Time(ms):140142145147149...180182
[0117] This smoothed series shows a clear upward trend, indicating that the number of logins is increasing over time.
[0118] (3) Drift judgment: Check whether there is any point where the maximum value of the left sequence is less than the minimum value of the right sequence. If this happens, then the data can be considered to have experienced a "sudden increase drift". The opposite situation, that is, the minimum value of the left sequence is greater than the maximum value of the right sequence, is considered a "sudden decrease drift".
[0119] For example, for the time series of user login times, if it is found that the median sequence of the most recent 30 days shows a clear upward trend, and the minimum value of the most recent 7 days is greater than the maximum value of the previous 23 days, it is determined that a sudden increase drift has occurred.
[0120] 3. Stability analysis method
[0121] The present invention uses the unit root test Augmented Dickey-Fuller (ADF) test to determine the stationarity of the time series. This is a commonly used statistical test to determine whether the time series is stationary, that is, whether the statistical properties of the series (such as mean and variance) remain unchanged over time. Below we will illustrate this process with data:
[0122] Assume that our user login times data is as follows:
[0123] Day: 1234567
[0124] Connections: 100105110105110115120
[0125] We perform an ADF test to determine whether the data for the last 1 day and the last 7 days are stationary.
[0126] Data from the last day: For the data from the last day, we only consider the value of the last day, which is 120. Since there is only one data point, the ADF test cannot be performed, so here we focus on the data from the last 7 days.
[0127] Data from the last 7 days: We use the data set 100, 105, 110, 105, 110, 115, 120 from the last 7 days to perform the ADF test.
[0128] (1) Perform ADF test: Perform ADF test on the data of the most recent day and the most recent 7 days. Assume that the result of performing ADF test is:
[0129] ADF test p value for the most recent day's data = 0.03;
[0130] The ADF test p value of the last 7 days of data is 0.02.
[0131] (2) Explanation of results: If the p-values of both time ranges are less than 0.05, the time series is considered to be stationary. For example, for the time series of user login times, if the ADF test p-values for the last 1 day and the last 7 days are 0.03 and 0.02 respectively, the series is considered to be stationary.
[0132] 2. Algorithm Selection Module
[0133] This module selects the appropriate AI detection algorithm based on data features and feature analysis results. Specific implementation:
[0134] (1) First, calculate the skewness value of the data distribution and preliminarily select the algorithm based on the skewness value:
[0135] Low skewness and high symmetry distribution (|skewness| < 0.5): use the median absolute difference (MAD);
[0136] Moderately skewed distribution (0.5≤|skewness|<1): Use box plot;
[0137] Highly skewed distribution (|skewness| ≥ 1): Use extreme value theory (EVT);
[0138] (2) Then, based on the results of periodicity, drift, and stationarity analysis, the primary algorithm is adjusted:
[0139] If there is a clear cycle:
[0140] For MAD and Boxplot, consider using periodic variants (such as periodic boxplots);
[0141] For EVT, consider performing seasonal decomposition before applying EVT;
[0142] If drift exists:
[0143] For all algorithms, consider introducing an adaptive threshold mechanism;
[0144] If the drift is significant, online learning algorithms or change point detection and segment modeling may be prioritized;
[0145] If non-stationary:
[0146] For MAD and Boxplot, consider differencing or detrending before applying the algorithm;
[0147] You may be inclined to choose algorithms such as ARIMA or LSTM that can handle non-stationary data.
[0148] Let's take an example: suppose we have a time series data of the number of user logins. We will analyze this data step by step and select the appropriate anomaly detection algorithm. For example, the time series of the number of user logins is output through the feature analysis module:
[0149] Periodicity: There is a clear daily cycle (24 hours);
[0150] Drift: A slight upward trend was detected;
[0151] Stationarity: The overall situation is not stationary, but it is close to stationary after removing the trend;
[0152] Data sample: one data point every 5 minutes for the last 30 days;
[0153] Then calculate the skewness value of the data distribution: calculate the skewness value: 0.8, and make a preliminary screening based on the skewness value (0.8): consider the box plot algorithm because it is moderately skewed. Next, consider the periodicity factor: due to the obvious daily cycle, consider the periodic box plot. On the other hand, consider the drift factor: a slight upward trend, consider introducing an adaptive threshold mechanism. Finally, consider the stationarity factor: overall non-stationary, consider detrending preprocessing. The final choice: periodic box plot + adaptive threshold + detrending preprocessing.
[0154] 3. Model Training Module
[0155] The present invention adopts the following technical architecture to implement model training:
[0156] Algorithm call: The algorithm factory mode is adopted to support dynamic loading of different anomaly detection algorithms. Standardized training and detection interfaces are implemented for each algorithm, so that different algorithms can be switched seamlessly. At the same time, parallel training of algorithms is supported to improve processing efficiency.
[0157] Periodicity processing: Design a periodicity detection function that can automatically identify periodic patterns in data. Support multiple periodic data processing methods, such as period segmentation, periodicity feature extraction, etc. For the detected period, implement data segmentation and aggregation processing.
[0158] Drift handling: Design drift detection function to identify changes in data distribution. Support dynamic adjustment of training window size to adapt to different degrees of drift. Implement incremental learning method so that the model can gradually adapt to changes in data.
[0159] Parameter optimization: Implement various parameter optimization methods, such as grid search, random search, etc. Support cross-validation to ensure the stability of parameter selection. At the same time, implement early stopping mechanism to avoid overfitting problems.
[0160] Model preservation: Design model serialization and deserialization mechanisms to support saving trained models to storage systems (such as Elasticsearch). Implement version control to allow saving and loading of different versions of models. At the same time, manage model metadata, such as training time, algorithms used, parameters, etc.
[0161] The model training module constitutes the core part of the entire time series data anomaly monitoring system. Its responsibility is to train a model that can accurately identify anomalies based on historical data. Taking the time series data of monitoring the number of user logins as an example, based on the aforementioned feature analysis of the number of user logins and the selected algorithm, the model begins the training process. It first segments the data into 24-hour cycles, and each segment of data may be as follows:
[0162]
[0163] For each time point, the system calculates a normal range based on historical data. For example, at 10:00 every day, the system may produce the following statistics:
[0164] Time: 10:00;
[0165] Minimum value: 150;
[0166] Lower quartile: 180;
[0167] Median: 200;
[0168] Upper quartile: 220;
[0169] Maximum value: 250;
[0170] The system will optimize the key parameters of the box plot algorithm, such as determining the best anomaly threshold multiple through cross-validation. Assuming that the final anomaly threshold multiple is 1.5, then for the time point of 10:00, the system will consider the number of connections below 115 (Q1-1.5*IQR) or above 285 (Q3+1.5*IQR) as abnormal.
[0171] In the box plot algorithm, outliers are usually defined by quartiles and interquartile ranges (IQR). Quartiles are divided into:
[0172] lower quartile (Q1);
[0173] Median(Q2);
[0174] Upper quartile (Q3).
[0175] The interquartile range (IQR) refers to the distance between the upper quartile Q3 and the lower quartile Q1, that is, IQR = Q3 - Q1.
[0176] Actually, Q1, Q2, and Q3 refer to the first quartile, the second quartile (which is also the median), and the third quartile, which divide the data set into four equal parts. That’s why they are called “quartiles.”
[0177] First Quartile (Q1): Also known as the lower quartile, it is the value that 25% of all values in the data set are less than or equal to. In other words, it separates the bottom 25% of the data set from the remaining 75%.
[0178] Second Quartile (Q2): Also known as the median, it is the value that 50% of all values in the data set are less than or equal to. It separates the first half of the data set from the second half.
[0179] Third Quartile (Q3): Also known as the upper quartile, it is the value that 75% of all values in the data set are less than or equal to. It separates the top 25% of the data set from the remaining 75%.
[0180] In this case, 115 (Q1-1.5*IQR): represents the lower anomaly threshold. Here, Q1 is the lower quartile, that is, the values in the data that are less than or equal to it account for 25% of the total, and IQR is the interquartile range. 1.5 is a constant used to define the range of outliers. Therefore, 115 is obtained by subtracting 1.5 times the IQR from Q1. If an observation is less than this value, it will be considered an outlier. 285 (Q3+1.5*IQR): represents the upper anomaly threshold. Similarly, Q3 is the upper quartile, that is, 75% of the values in the data are less than or equal to it. By adding 1.5 times the IQR to Q3, the value of 285 is obtained. If an observation is greater than this value, it will also be considered an outlier.
[0181] To handle slight drifts in data, the system uses an incremental learning approach. It updates the model with the latest data regularly (e.g., weekly) and gradually adjusts the normal range at each time point.
[0182] After training is completed, the system will serialize the model and save it to Elasticsearch, while recording the model's metadata, such as training time, algorithm used, parameters, etc. In this way, the real-time detection module can load the latest model at any time to perform anomaly detection.
[0183] Finally, the system will evaluate the trained model. It will use the reserved test data set to test the model's precision, recall and other indicators. Assume that the evaluation results show that the model has achieved 85% precision and 83% recall on the test set, which indicates that the model has good performance and can effectively identify abnormalities in the number of user logins.
[0184] The test dataset is usually split from the original dataset to independently evaluate the performance of the model. When splitting the dataset, a certain ratio is usually followed, such as 80% training data, 10% validation data, and 10% test data; or 70% training data, 15% validation data, and 15% test data. The specific split ratio depends on the size of the dataset and the needs of the project.
[0185] Through this comprehensive and flexible training process, the model training module can create a high-performance anomaly detection model, laying a solid foundation for real-time monitoring of abnormal fluctuations in time series data. This enables the operation and maintenance team to be sensitive to potential anomalies, such as sudden surges or unexpected drops in data points, and then respond quickly and take effective measures to ensure smooth business processes and stable data flows.
[0186] 4. Real-time Detection Module
[0187] The real-time detection module is a key component of the entire system and is responsible for anomaly detection of real-time data. It continuously monitors data indicators based on the trained model and identifies potential anomalies in a timely manner. The following is a detailed description of the real-time detection module:
[0188] 1. Data access
[0189] The real-time detection module first receives monitoring data in real time through a message queue (such as Kafka). For the case of user login times, the system may receive data once a minute in the following format:
[0190] Timestamp: 2024-07-12 14:30:00;
[0191] Metric name: user_login;
[0192] Value: 230;
[0193] 2. Data preprocessing
[0194] After receiving the data, the module will perform preliminary data preprocessing:
[0195] Data formatting: Convert timestamps to a standard format to ensure consistency with training data.
[0196] Data validation: Checks whether the value is within a reasonable range, for example, the number of connections should not be negative.
[0197] Missing value handling: If the data for a certain minute is missing, it may be filled with the last valid value.
[0198] 3. Model loading
[0199] The module loads the latest detection model from the storage system (such as Elasticsearch) periodically (such as every hour). This ensures that the system always uses the latest model that adapts to the current data characteristics for detection.
[0200] 4. Feature extraction
[0201] Based on the requirements of the loaded model, the system extracts the necessary features. For the periodic box plot model, the main feature is time. The system maps the current time (14:30) to the corresponding time period in the model.
[0202] 5. Anomaly Detection
[0203] The preprocessed data is input into the model for anomaly detection. The model will determine whether the current value is abnormal based on the normal range calculated from historical data. For example:
[0204] Time: 14:30;
[0205] Current value: 230;
[0206] Normal range: 180-250;
[0207] Judgment result: normal.
[0208] If the current value is outside the normal range, the system will mark it as abnormal.
[0209] 6. Result output
[0210] The test results are written to the storage system (such as Elasticsearch) in real time, and the format may be as follows:
[0211] Timestamp: 2024-07-12 14:30:00;
[0212] Metric name: user_login;
[0213] Current value: 230;
[0214] Expected range: 180-250;
[0215] Status: Normal.
[0216] If an anomaly is detected, the system will generate a structured warning message:
[0217] Alert ID: ALERT-20240712-001;
[0218] Time: 2024-07-12 14:35:00;
[0219] Metrics: Number of user logins;
[0220] Current value: 280;
[0221] Expected range: 180-250;
[0222] Severity: Moderate;
[0223] Description: The number of user logins exceeds the normal range, which may lead to resource exhaustion;
[0224] Recommended Action: Check the user account status.
[0225] Through this comprehensive and real-time detection process, the real-time detection module can quickly identify abnormal conditions in time series data, such as sudden changes in the number of user logins, and provide accurate and immediate warning information to the operations team. This helps them quickly locate and respond to potential challenges, thereby ensuring the smooth operation of the business system and optimizing overall performance.
[0226] It should be noted that although several modules of the time series data intelligent anomaly detection system based on a large model are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for concretization.
[0227] Based on the same inventive concept, the present invention also proposes a time series data intelligent anomaly detection method based on a large model. Figure 2 As shown, the method includes:
[0228] 1. Receive business time series data, use the moving average method to identify the periodicity of the time series data, use the median filter method to detect the drift of the time series data, and use the ADF test to determine the stationarity of the time series data.
[0229] Use the following steps to identify the periodicity of time series data:
[0230] Use the moving average method to extract trend components from time series data;
[0231] Subtract the original sequence of time series data from the trend component to obtain the residual sequence;
[0232] Perform cyclic shift on the residual sequence and perform vector point multiplication with the original sequence to calculate the autocorrelation sequence;
[0233] Extract the local maximum peak of the autocorrelation sequence, and take the horizontal axis interval as the period T;
[0234] If the autocorrelation value at period T is greater than a given threshold, significant periodicity is considered to exist.
[0235] Use the following steps to detect drift in time series data:
[0236] Select an appropriate window size, extract the median within the window and perform smoothing;
[0237] Check whether the smoothed series shows a strictly increasing or decreasing trend;
[0238] Check whether there is any point where the maximum value of the sequence on the left is less than the minimum value of the sequence on the right; if so, it is considered that the data has experienced a sudden increase drift; check whether there is any point where the minimum value of the sequence on the left is greater than the maximum value of the sequence on the right; if so, it is considered that the data has experienced a sudden decrease drift.
[0239] 2. Select the absolute median difference, box plot or extreme value theory according to the skewness value of the data distribution, and adjust the parameters of the selected anomaly detection algorithm based on the analysis results of periodicity, drift and stationarity.
[0240] Use the following steps to select an appropriate anomaly detection algorithm and tune it:
[0241] (1) Calculate the skewness value of the data distribution and preliminarily select the algorithm based on the skewness value:
[0242] |Skewness|<0.5: use absolute median difference; 0.5≤|Skewness|<1: use box plot; |Skewness|≥1: use extreme value theory;
[0243] (2) According to the analysis results of periodicity, drift and stability, the preliminary algorithm is adjusted:
[0244] If cycles are evident: for absolute median difference and box plots, consider using a cyclical variant; for extreme value theory, consider performing a seasonal decomposition before applying extreme value theory;
[0245] If drift exists: For all algorithms, consider introducing an adaptive threshold mechanism; if drift is significant, online learning algorithms or change point detection and segmentation modeling may be prioritized;
[0246] If non-stationary: For absolute median difference and box plots, consider differencing or detrending before applying the algorithm; you may be tempted to choose an algorithm such as ARIMA or LSTM that can handle non-stationary data.
[0247] 3. Adopt the algorithm factory mode to support dynamic loading and parallel training of different anomaly detection algorithms; for periodic data, enhance model training through segmentation and aggregation processing; for drifting data, adopt incremental learning methods to make the model gradually adapt to data changes; use parameter optimization methods to ensure optimal model performance.
[0248] The following technical architecture is used to implement model training:
[0249] Adopting the algorithm factory mode, it supports dynamic loading of different anomaly detection algorithms; implements standardized training and detection interfaces for each algorithm; and supports parallel training of algorithms;
[0250] Design cycle detection function to automatically identify periodic patterns in data; support multiple methods for processing periodic data; implement segmentation and aggregation of data for detected cycles;
[0251] Design drift detection function to identify changes in data distribution; support dynamic adjustment of training window size; implement incremental learning method to make the model gradually adapt to data changes;
[0252] Implement multiple parameter optimization methods; support cross-validation to ensure the stability of parameter selection; and implement early stopping mechanism;
[0253] Design model serialization and deserialization mechanisms to support saving trained models to storage systems; implement version control to allow saving and loading of different versions of models; and manage model metadata.
[0254] 4. Continuously receive real-time time series data, load the trained model after preprocessing for anomaly detection, and output the results immediately once an anomaly is detected.
[0255] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0256] Based on the above invention concept, Figure 3 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, the aforementioned large model-based intelligent anomaly detection method for time series data is implemented.
[0257] Based on the aforementioned inventive concept, the present invention also proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned large model-based intelligent anomaly detection method for time series data.
[0258] The time series data intelligent anomaly detection system and method based on a large model proposed in the present invention have the following highlights:
[0259] 1. Intelligent analysis and dynamic adaptation: The system automatically identifies the periodicity, drift and stability characteristics of the data, intelligently selects and adjusts the anomaly detection algorithm to ensure that the model can capture the inherent laws of the data, thereby improving the accuracy and flexibility of detection.
[0260] 2. Efficient model training and updating: Supports dynamic loading and parallel training of algorithms, and adopts segmented aggregation, incremental learning and parameter optimization strategies to ensure that the model quickly adapts to data changes, maintains high accuracy and reduces maintenance costs.
[0261] 3. Real-time anomaly detection and immediate response: The real-time detection module continuously monitors data, loads the trained model for anomaly detection, responds immediately and outputs results, achieving early warning of abnormal events and accelerating decision-making.
[0262] 4. High precision and wide applicability: Compared with traditional threshold monitoring, the system has higher anomaly detection accuracy and recall rate, can perform automatic inspections around the clock, adapt to various business scenarios, and provide strong support for business stability and sustainable development.
[0263] 5. Cost-effectiveness and process optimization: Through automated and intelligent anomaly detection, the need for manual monitoring is reduced, operating costs are reduced, and the efficiency and reliability of business processes are improved, especially in areas such as financial transactions, network security, and production control that require high reliability and responsiveness.
[0264] In summary, the system can provide modern enterprises with more efficient and reliable time series data anomaly monitoring solutions, ensuring the smooth operation and continuous optimization of the business.
[0265] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.
[0266] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.
Claims
1. An intelligent anomaly detection system for time series data based on a large model, characterized in that: The system includes: The feature analysis module is used to identify the periodicity of time series data using the moving average method, detect the drift of time series data using the median filter method, and determine the stability of time series data using the ADF test; The algorithm selection module is used to select the absolute median difference, box plot or extreme value theory according to the skewness value of the data distribution, and adjust the parameters of the selected anomaly detection algorithm according to the analysis results of periodicity, drift and stationarity; Model training module, which uses the algorithm factory mode to support dynamic loading and parallel training of different anomaly detection algorithms; for periodic data, segmentation and aggregation processing are used to enhance model training; for drifting data, incremental learning methods are used to gradually adapt the model to data changes; parameter optimization methods are used to ensure optimal model performance; The real-time detection module is used to continuously receive real-time time series data, load the trained model after preprocessing for anomaly detection, and output the results immediately once an anomaly is detected.
2. The time series data intelligent anomaly detection system based on a large model according to claim 1 is characterized in that: Use the following steps to identify the periodicity of time series data: Use the moving average method to extract trend components from time series data; Subtract the original sequence of time series data from the trend component to obtain the residual sequence; Perform cyclic shift on the residual sequence and perform vector point multiplication with the original sequence to calculate the autocorrelation sequence; Extract the local maximum peak of the autocorrelation sequence, and take the horizontal axis interval as the period T; If the autocorrelation value at period T is greater than a given threshold, significant periodicity is considered to exist.
3. The time series data intelligent anomaly detection system based on a large model according to claim 1 is characterized in that: Use the following steps to detect drift in time series data: Select an appropriate window size, extract the median within the window and perform smoothing; Check whether the smoothed series shows a strictly increasing or decreasing trend; Check whether there is any point where the maximum value of the sequence on the left is less than the minimum value of the sequence on the right; if so, it is considered that the data has experienced a sudden increase drift; check whether there is any point where the minimum value of the sequence on the left is greater than the maximum value of the sequence on the right; if so, it is considered that the data has experienced a sudden decrease drift.
4. The time series data intelligent anomaly detection system based on a large model according to claim 1 is characterized in that: Use the following steps to select an appropriate anomaly detection algorithm and tune it: (1) Calculate the skewness value of the data distribution and preliminarily select the algorithm based on the skewness value: |Skewness|<0.5: use absolute median difference; 0.5≤|Skewness|<1: use box plot; |Skewness|≥1: use extreme value theory; (2) According to the analysis results of periodicity, drift and stability, the preliminary algorithm is adjusted: If cycles are evident: for absolute median difference and box plots, consider using a cyclical variant; for extreme value theory, consider performing a seasonal decomposition before applying extreme value theory; If drift exists: For all algorithms, consider introducing an adaptive threshold mechanism; if drift is significant, online learning algorithms or change point detection and segmentation modeling may be prioritized; If non-stationary: For absolute median difference and box plots, consider differencing or detrending before applying the algorithm; you may be tempted to choose an algorithm such as ARIMA or LSTM that can handle non-stationary data.
5. The time series data intelligent anomaly detection system based on a large model according to claim 1 is characterized in that: The following technical architecture is used to implement model training: Adopting the algorithm factory mode, it supports dynamic loading of different anomaly detection algorithms; implements standardized training and detection interfaces for each algorithm; and supports parallel training of algorithms; Design cycle detection capabilities to automatically identify periodic patterns in data; Supports multiple methods for processing periodic data; for the detected period, implements data segmentation and aggregation processing; Design drift detection capabilities to identify changes in data distribution; Support dynamic adjustment of training window size; implement incremental learning method to make the model gradually adapt to data changes; Implement various parameter optimization methods; Support cross-validation to ensure the stability of parameter selection; also implement early stopping mechanism; Design model serialization and deserialization mechanisms to support saving trained models to storage systems; Implement version control, allowing different versions of the model to be saved and loaded; also manage the model's metadata.
6. A method for intelligent anomaly detection of time series data based on a large model, characterized in that: The method includes: Use the moving average method to identify the periodicity of time series data, use the median filter method to detect the drift of time series data, and use the ADF test to determine the stationarity of time series data; Select the absolute median difference, box plot or extreme value theory according to the skewness value of the data distribution, and adjust the parameters of the selected anomaly detection algorithm based on the analysis results of periodicity, drift and stationarity; The algorithm factory mode is adopted to support dynamic loading and parallel training of different anomaly detection algorithms. For periodic data, segmentation and aggregation processing are used to enhance model training. For drifting data, incremental learning methods are used to gradually adapt the model to data changes. Parameter optimization methods are used to ensure that the model performance is optimal. Continuously receive real-time time series data, load the trained model after preprocessing for anomaly detection, and output the results immediately once an anomaly is detected.
7. The method for intelligent anomaly detection of time series data based on a large model according to claim 6 is characterized in that: Use the following steps to identify the periodicity of time series data: Use the moving average method to extract trend components from time series data; Subtract the original sequence of time series data from the trend component to obtain the residual sequence; Perform cyclic shift on the residual sequence and perform vector point multiplication with the original sequence to calculate the autocorrelation sequence; Extract the local maximum peak of the autocorrelation sequence, and take the horizontal axis interval as the period T; If the autocorrelation value at period T is greater than a given threshold, significant periodicity is considered to exist.
8. The method for intelligent anomaly detection of time series data based on a large model according to claim 6 is characterized in that: Use the following steps to detect drift in time series data: Select an appropriate window size, extract the median within the window and perform smoothing; Check whether the smoothed series shows a strictly increasing or decreasing trend; Check whether there is any point where the maximum value of the sequence on the left is less than the minimum value of the sequence on the right; if so, it is considered that the data has experienced a sudden increase drift; check whether there is any point where the minimum value of the sequence on the left is greater than the maximum value of the sequence on the right; if so, it is considered that the data has experienced a sudden decrease drift.
9. The method for intelligent anomaly detection of time series data based on a large model according to claim 6, characterized in that: Use the following steps to select an appropriate anomaly detection algorithm and tune it: (1) Calculate the skewness value of the data distribution and preliminarily select the algorithm based on the skewness value: |Skewness|<0.5: use absolute median difference; 0.5≤|Skewness|<1: use box plot; |Skewness|≥1: use extreme value theory; (2) According to the analysis results of periodicity, drift and stability, the preliminary algorithm is adjusted: If cycles are evident: for absolute median difference and box plots, consider using a cyclical variant; for extreme value theory, consider performing a seasonal decomposition before applying extreme value theory; If drift exists: For all algorithms, consider introducing an adaptive threshold mechanism; if drift is significant, online learning algorithms or change point detection and segmentation modeling may be prioritized; If non-stationary: For absolute median difference and box plots, consider differencing or detrending before applying the algorithm; you may be tempted to choose an algorithm such as ARIMA or LSTM that can handle non-stationary data.
10. The method for intelligent anomaly detection of time series data based on a large model according to claim 6, characterized in that: The following technical architecture is used to implement model training: Adopting the algorithm factory mode, it supports dynamic loading of different anomaly detection algorithms; implements standardized training and detection interfaces for each algorithm; and supports parallel training of algorithms; Design cycle detection capabilities to automatically identify periodic patterns in data; Supports multiple methods for processing periodic data; for the detected period, implements data segmentation and aggregation processing; Design drift detection capabilities to identify changes in data distribution; Support dynamic adjustment of training window size; implement incremental learning method to make the model gradually adapt to data changes; Implement various parameter optimization methods; Support cross-validation to ensure the stability of parameter selection; also implement early stopping mechanism; Design model serialization and deserialization mechanisms to support saving trained models to storage systems; Implement version control, allowing different versions of the model to be saved and loaded; also manage the model's metadata.
11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 6 to 10 is implemented.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 6 to 10.
Citation Information
Cited By
Agent intelligent customer service and anomaly detection system
CN119624467A
Navigation-aid light monitoring method and system based on fail-safe
CN119629822A
Intelligent control method and device based on processing process management and chip
CN119644895A
Intelligent Control Method, Device and Chip Based on Processing Process Management
CN119644895B
