Abnormal data prediction method and system based on dynamic threshold and pattern mining

Abstract

The invention discloses an abnormal data prediction method and system based on a dynamic threshold value and pattern mining, and the method comprises the steps: carrying out the mining of an association rule and a sequence pattern of historical monitoring data, recognizing the association relationship among a plurality of monitoring indexes under different data collection sources, and extracting a frequently appearing sequence pattern; setting a dynamic threshold range based on the mean value and the standard deviation of the historical monitoring data; dynamically adjusting the threshold range according to the real-time operation state and the change trend of the historical monitoring data; and through matching the real-time data with the historical mode, predicting a potential anomaly, and giving out an early warning. According to the method and the system, through adaptive dynamic threshold and abnormal mode mining, intelligent prediction and early warning of anomalies in a data acquisition process are realized, and post-processing is converted into beforehand early warning; through real-time monitoring and model analysis, the system can detect potential anomalies in advance, optimize data quality, improve operation and maintenance efficiency and reduce manual intervention.

Description

Technical Field

[0001] The present invention relates to the field of data collection and monitoring, and in particular to an abnormal data prediction method and system based on dynamic threshold and pattern mining. Background Art

[0002] In the process of data collection and monitoring, the system often encounters complex and changing environments. For example, drastic changes in system load or data flow, increased real-time requirements, and complex correlations between different indicators all pose severe challenges to traditional monitoring systems. If these problems cannot be warned or resolved in a timely manner, they may lead to abnormal inflows of business system data, seriously affecting the normal operation of business processes. These challenges are as follows:

[0003] 1. The lag of traditional monitoring

[0004] Traditional monitoring systems can usually detect abnormal events only after they have already occurred. For example, during the data collection process, if data loss or data anomalies occur, they are usually not discovered until the problem has entered the business system. This brings potential risks to the business system because these abnormal data may have been processed or passed to other systems, leading to chain problems. Repairs are delayed and complicated, especially in the flow of data across systems or processes, which makes problem solving difficult and costly.

[0005] 2. Limitations of fixed threshold monitoring

[0006] Many traditional monitoring systems rely on preset fixed thresholds to detect abnormal data. However, these thresholds may no longer apply as system load, data traffic, or operating conditions change. For example, in the case of a surge or decrease in system traffic, fixed thresholds may lead to false positives or false negatives, where normal data is marked as abnormal or true abnormal data is not identified. Such misjudgments result in unnecessary waste of resources and may even cause business interruptions.

[0007] 3. Complex abnormal patterns are difficult to capture

[0008] Traditional monitoring methods mainly rely on simple rules or single indicators, which makes it difficult for them to capture abnormal patterns involving complex interactions between multiple indicators or specific time series. For example, some anomalies may require combining multiple data sources or rely on trends over a longer time frame to be identified, which traditional monitoring often cannot do. This limitation will prevent some potential problems from being discovered in the early stages, leading to greater business risks.

[0009] 4. High dependence on human intervention

[0010] Although the monitoring system can capture abnormal situations, in many cases manual intervention is still required to troubleshoot and resolve the problem. Frequent manual intervention not only increases operation and maintenance costs, but also easily leads to human errors, which further expand the problem. Summary of the invention

[0011] In order to solve the above-mentioned problems existing in traditional monitoring systems, the present invention provides an abnormal data prediction method and system based on dynamic thresholds and pattern mining. Through adaptive dynamic thresholds and abnormal pattern mining, intelligent prediction and early warning of abnormalities in the data collection process are realized, transforming from "post-processing" to "pre-warning"; through real-time monitoring and model analysis, the system can detect potential anomalies in advance, optimize data quality, improve operation and maintenance efficiency, and reduce manual intervention.

[0012] To achieve the above purpose, the present invention adopts the following technical solutions:

[0013] In one embodiment of the present invention, a method for predicting abnormal data based on dynamic threshold and pattern mining is proposed, the method comprising:

[0014] By mining the association rules and sequence patterns of historical monitoring data, we can identify the associations between multiple monitoring indicators under different data collection sources and extract frequently occurring sequence patterns.

[0015] Set dynamic threshold ranges based on the mean and standard deviation of historical monitoring data; dynamically adjust threshold ranges based on real-time operating status and changing trends of historical monitoring data;

[0016] By matching real-time data with historical patterns, potential anomalies can be predicted and early warnings can be issued.

[0017] Furthermore, association rule mining is implemented through the following steps:

[0018] Find frequently occurring item sets by scanning historical monitoring data;

[0019] Based on frequently occurring item sets, association rules are generated and their support and confidence are calculated.

[0020] Furthermore, sequential pattern mining is achieved through the following steps:

[0021] Divide historical monitoring data into multiple time windows in chronological order, and extract the monitoring indicator values ​​in the time windows;

[0022] Analyze the monitoring indicator values ​​in different time windows and generate candidate sequence patterns;

[0023] By scanning these candidate sequence patterns, the sequence patterns that frequently appear in multiple time windows are found;

[0024] Extract frequently occurring sequential patterns from time series.

[0025] In one embodiment of the present invention, an abnormal data prediction system based on dynamic threshold and pattern mining is also proposed, and the system includes:

[0026] The association rule and sequence pattern mining module is used to identify the association relationship between multiple monitoring indicators under different data collection sources and extract frequently occurring sequence patterns through association rule and sequence pattern mining of historical monitoring data;

[0027] Adaptive dynamic threshold module, used to set dynamic threshold range based on the mean and standard deviation of historical monitoring data; dynamically adjust threshold range according to real-time operation status and change trend of historical monitoring data;

[0028] The predictive warning module is used to predict potential anomalies and issue warnings by matching real-time data with historical patterns.

[0029] Furthermore, association rule mining is implemented through the following steps:

[0030] Find frequently occurring item sets by scanning historical monitoring data;

[0031] Based on frequently occurring item sets, association rules are generated and their support and confidence are calculated.

[0032] Furthermore, sequential pattern mining is achieved through the following steps:

[0033] Divide historical monitoring data into multiple time windows in chronological order, and extract the monitoring indicator values ​​in the time windows;

[0034] Analyze the monitoring indicator values ​​in different time windows and generate candidate sequence patterns;

[0035] By scanning these candidate sequence patterns, the sequence patterns that frequently appear in multiple time windows are found;

[0036] Extract frequently occurring sequential patterns from time series.

[0037] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned abnormal data prediction method based on dynamic thresholds and pattern mining is designed.

[0038] In one embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program for executing an abnormal data prediction method based on dynamic thresholds and pattern mining.

[0039] Beneficial effects:

[0040] 1. The present invention has achieved a strategic transformation from the traditional "post-event response" to "pre-event insight" by introducing an intelligent predictive early warning mechanism. The system can predict potential abnormal situations in advance based on historical monitoring data mining, issue warning notifications in advance, avoid the expansion of problems or system crashes, and ensure business continuity and high availability of the system.

[0041] 2. The present invention uses an advanced adaptive dynamic threshold adjustment algorithm, and the system can dynamically adjust the monitoring threshold according to the current operating load and environment. This flexible and accurate threshold adjustment mechanism can effectively reduce false positives and false negatives, and ensure that efficient anomaly detection capabilities are maintained under different loads.

[0042] 3. By using cutting-edge technologies such as association rule mining and time series pattern recognition, the system can automatically extract complex abnormal patterns from massive historical monitoring data. This not only greatly expands the system's data insight capabilities, but also accurately captures the potential connections and trend changes between various monitoring indicators, builds a high-dimensional anomaly detection framework, and forms a more comprehensive early warning system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic flow chart of an abnormal data prediction method based on dynamic threshold and pattern mining according to an embodiment of the present invention;

[0044] Figure 2 It is a schematic diagram of the structure of an abnormal data prediction system based on dynamic threshold and pattern mining of the present invention;

[0045] Figure 3 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0046] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and design the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0047] Those skilled in the art will appreciate that the embodiments of the present invention may be designed as a system, system, device, method or computer program product. Therefore, the present disclosure may be specifically designed in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0048] According to the implementation of the present invention, a method for predicting abnormal data based on dynamic thresholds and pattern mining is proposed. Through adaptive dynamic thresholds and abnormal pattern mining, intelligent prediction and early warning of abnormalities in the data collection process are realized, transforming from "post-processing" to "pre-warning". Through real-time monitoring and model analysis, potential abnormalities can be detected in advance, data quality can be optimized, operation and maintenance efficiency can be improved, and manual intervention can be reduced.

[0049] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0050] The abnormal data prediction method based on dynamic threshold and pattern mining of the present invention comprises:

[0051] 1. Association rules and sequence pattern mining

[0052] By mining the association rules and sequence patterns of historical monitoring data, the complex associations and change trends between multiple monitoring indicators can be identified. By using association rule mining technology, the associations between multiple monitoring indicators under different data collection sources can be identified, and frequently occurring sequence patterns, i.e., abnormal patterns, can be extracted. By comparing the current collected data with known abnormal patterns in real time, the system can detect potential risks in advance and issue warnings to ensure that the problem does not spread further. In this way, the system can capture more complex abnormal patterns, thereby greatly improving the accuracy and foresight of abnormality detection.

[0053] 2. Introducing dynamic thresholds

[0054] Traditional monitoring systems usually rely on static thresholds, which can easily lead to false alarms or missed alarms when the system load or data traffic changes. To solve this problem, this solution sets a dynamic threshold range by calculating the mean and standard deviation of historical monitoring data. The system adaptively adjusts the threshold range for anomaly detection based on the real-time operating status and the changing trend of historical monitoring data. When the load increases or the environment changes, the system automatically updates the threshold range to adapt to the new operating conditions, avoiding false alarms or missed alarms, thereby improving the accuracy and reliability of monitoring.

[0055] 3. Predictive warning based on pattern mining

[0056] The system identifies potential anomalies through real-time pattern matching and predictive analysis. During the data collection process, the system continuously monitors multi-dimensional collection data. By capturing the sequence patterns between different monitoring indicators and combining the analysis of historical monitoring data, the system can detect abnormal patterns in advance and predict possible collection anomalies in the future. When potential risks are predicted, the system will issue early warning signals for operation and maintenance personnel or the system to handle to avoid the impact of abnormalities on the business. This pattern mining method can effectively identify complex abnormal patterns and improve the accuracy and timeliness of problem identification.

[0057] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0058] In order to explain the above abnormal data prediction method based on dynamic threshold and pattern mining more clearly, a specific embodiment is used for illustration below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0059] Example:

[0060] like Figure 1 As shown, the specific implementation steps are as follows:

[0061] 1. Association rules and sequential pattern mining

[0062] The purpose of association rules and sequence pattern mining is to find patterns that reflect system anomalies by analyzing the associations between different monitoring indicators in historical monitoring data. The specific implementation process is as follows:

[0063] 1. Data preparation:

[0064] Data preparation is important because it provides standardized input for subsequent pattern mining.

[0065] (1) Data Collection

[0066] Collect historical monitoring data from different data collection sources (such as server monitoring systems, network traffic monitoring systems, etc.), including key indicators at each time point, such as: data traffic, CPU usage, memory usage, response time, network latency, etc.

[0067] (2) Data cleaning

[0068] Clean the collected data to remove missing values, outliers, and duplicate data. For example, if the data flow indicator is missing at a certain point in time, you can fill in or delete the data to handle it.

[0069] (3) Data standardization and normalization

[0070] Standardize and normalize the data so that the data scales of different features are consistent. For example, data flow is in MB / s, while response time is in seconds.

[0071] (4) Structural arrangement

[0072] The cleaned data is organized into a structured table format (usually a two-dimensional table), where each row represents an observation record at a time point and each column represents a different monitoring indicator. This makes it easier to perform subsequent association rules and pattern mining.

[0073] The sample data is shown in Table 1 below:

[0074] Table 1

[0075]

[0076] 2. Association rule mining:

[0077] The goal of association rule mining is to identify the relationship between various monitoring indicators. The specific implementation process is as follows:

[0078] Algorithm introduction: The algorithm for generating association rules based on frequent item sets (existing algorithm) is used to generate association rules.

[0079] Frequent item set generation: Find frequently occurring item sets (i.e., situations where multiple indicator combinations occur frequently) by scanning the historical monitoring data processed in step 1. For the monitoring indicators in this example, assuming that we consider "data flow > 60MB / s" as one item and "CPU usage > 75%" as another item, by scanning the historical monitoring data, we can find which items often occur at the same time.

[0080] Association rule generation: Based on frequently occurring item sets, association rules are generated and their support and confidence are calculated.

[0081] For example, the support of the association rule "If the data flow rate > 60MB / s, the network delay > 100ms" refers to the frequency of the rule appearing in the historical monitoring data, and the confidence refers to the probability that the network delay is greater than 100ms when the data flow rate is greater than 60MB / s.

[0082] (1) Convert historical monitoring data into binary format. For example, convert "data flow > 60MB / s" into 1 (indicating that the condition is met), otherwise it is 0. In this way, each monitoring indicator is converted into a binary representation of whether the condition is met or not.

[0083] (2) Generate frequent item sets by scanning all possible item sets. For example, "data flow > 60MB / s" and "CPU usage > 75%" may be satisfied at the same time, generating a frequent item set.

[0084] (3) Based on these frequent item sets, association rules are generated and the support (frequency of occurrence of the rule) and confidence (probability of the result occurring when the conditions are met) of the rules are calculated.

[0085] In summary, association rule mining yields the following rule examples:

[0086] Rule: If data traffic > 60MB / s and CPU usage > 75%, network latency > 100ms.

[0087] Support = (number of records that satisfy both the antecedent and consequent) / total number of records.

[0088] Support refers to the number of records with data flow > 60MB / s, CPU usage > 75%, and network latency > 100ms occurring simultaneously, as a proportion of the entire data set.

[0089] For example, if there are 100 records in the historical monitoring data, and 30 records meet the conditions of "data flow > 60MB / s, CPU usage > 75%, and network delay > 100ms", the support is calculated as:

[0090] Support rate = 30 / 100 = 30%.

[0091] Support: 30% (the rule holds true 30% of the time).

[0092] Confidence = (number of records where both the antecedent and the consequent are satisfied) / number of records where the antecedent is satisfied.

[0093] Confidence refers to the probability that the network delay is greater than 100ms when "data traffic > 60MB / s and CPU usage > 75%".

[0094] Assume that among the records that meet the conditions of "data flow>60MB / s and CPU usage>75%", there are 40 records that meet the conditions (excluding network latency), and 32 of them also meet the conditions of "network latency>100ms".

[0095] Confidence level = 32 / 40 = 80%.

[0096] Confidence level: 80% (when the traffic exceeds 60MB / s and the CPU usage is greater than 75%, the network delay will exceed 100ms in 80% of the cases).

[0097] 3. Sequential pattern mining:

[0098] The purpose of sequence pattern mining is to discover the changing trends of different monitoring indicators at multiple time points and help identify complex time series anomalies. This process focuses on the changes in data in the time dimension and whether these changes conform to specific anomaly patterns. The specific implementation process is as follows:

[0099] (1) Data preparation

[0100] Make sure each data point has a clear timestamp and is arranged in chronological order. Each data point represents the value of the monitoring indicator within a certain period of time. For example, assume that the monitored data includes: data traffic, CPU usage, network latency, and response time. We collect the data in Table 2 below.

[0101] Table 2

[0102]

[0103]

[0104] (2) Sequential pattern mining algorithm

[0105] The GSP algorithm is a classic sequential pattern mining algorithm that aims to find frequent sequential patterns through time series analysis. The main steps of the algorithm are as follows:

[0106] The data is divided into multiple time windows (for example, one window per minute) in chronological order, and the monitoring indicator values ​​in the time windows are extracted. For example, the data in the time period from 12:00:00 to 12:04:00 can be divided into five time windows.

[0107] Analyze the values ​​of monitoring indicators in different time windows to generate possible sequence patterns. For example, if "data traffic continues to increase" and "CPU usage increases" appear at the same time, this can form a candidate sequence pattern. Perform frequent item set analysis on these candidate sequence patterns to find sequence patterns that frequently appear in multiple time windows.

[0108] The system scans all time series to find frequently occurring patterns. For example, suppose that every time "data flow > 60MB / s" and "CPU usage > 75%", network latency and response time increase, and this pattern occurs frequently over multiple time periods.

[0109] In summary, sequential pattern mining yields the following sequential pattern examples:

[0110] For 5 consecutive minutes, data traffic continued to rise and CPU usage gradually increased, resulting in a significant increase in network latency and response time.

[0111] (3) Generate abnormal sequence patterns

[0112] Eventually, the system will extract frequent sequence patterns from the time series. For example, if the system finds that the data traffic gradually increases, the CPU usage increases, and the network delay and response time increase frequently in multiple time periods, this can be used as one of the abnormal warning patterns.

[0113] 2. Introducing dynamic thresholds

[0114] The main purpose of introducing dynamic thresholds is to dynamically adjust the threshold range of the monitoring system through the mean and standard deviation of historical monitoring data to ensure that the system can adapt to load changes and detect abnormal situations in a timely manner. For example, during peak hours, data traffic and CPU usage are usually high. The system will automatically adjust the threshold range based on the peak data to allow a higher fluctuation range. During off-peak hours when the system load is light, the threshold range will automatically narrow to ensure that smaller abnormal fluctuations are detected. The specific implementation process is as follows:

[0115] 1. Standard Deviation

[0116] Standard deviation is an important indicator for measuring data volatility. It reflects the degree of deviation of a set of data points from the mean. The larger the standard deviation, the more drastic the data fluctuations; the smaller the standard deviation, the relatively stable the data. There is a fixed calculation formula for standard deviation.

[0117] Standard deviation calculation formula:

[0118]

[0119] Where: x i represents each data point; μ represents the mean of the data; n represents the number of data points.

[0120] Assume that the monitored data traffic (unit: MB / s) is as follows:

[0121] 40,50,55,45,60MB / s;

[0122] Average value: (40+50+55+45+60) / 5=50MB / s;

[0123] This means that the average data flow rate is 50MB / s.

[0124] Compute the difference between each data point and the mean:

[0125] (40-50)2=100, (50-50)2=0, (55-50)2=25, (45-50)2=25, (60-50)2=100.

[0126] Calculate the standard deviation:

[0127]

[0128] The standard deviation is 7.07MB / s, which indicates the degree of fluctuation of data traffic during this time period.

[0129] 2. Setting of dynamic threshold range

[0130] Based on the mean and standard deviation, a dynamic threshold range can be set. The dynamic threshold range is usually set according to the following formula:

[0131] Threshold range = [A-2B, A+2B];

[0132] Among them, A is the mean of historical monitoring data, and B is the standard deviation.

[0133] Example data threshold setting:

[0134] Threshold range = [50-2×7.07, 50+2×7.07] = [35.86MB / s, 64.14MB / s].

[0135] Data flow within this range (35.86MB / s to 64.14MB / s) is considered a normal fluctuation. If the data flow exceeds this range (for example, below 35.86MB / s or above 64.14MB / s), the system will identify it as abnormal and may issue a warning.

[0136] 3. Dynamically adjust threshold range based on new data points

[0137] When the system load changes (such as peak and off-peak periods), the system's operating data will continue to change. The real-time monitoring system needs to dynamically adjust the threshold range based on the new data points to ensure that the threshold range can follow the changes in the current load and adapt to different operating conditions.

[0138] New data points are the latest monitoring data collected by the system in real time in each time period (such as every minute, every second). These data include key indicators such as network traffic and CPU usage in the current time period. As new data points are introduced, the system will continuously update the mean and standard deviation, thereby dynamically adjusting the threshold range.

[0139] Example data:

[0140] In the next 5 minutes, the system continues to collect the following data traffic:

[0141] 55,65,70,50,45MB / s.

[0142] The mean and standard deviation can be recalculated based on this new set of data points. The formula in step 2 above can be used to calculate the mean to be 57MB / s and the standard deviation to be 9.27MB / s. Based on the new mean and standard deviation, the dynamic threshold will be adjusted to:

[0143] Threshold range = [57-2×9.27, 57+2×9.27] = [38.46MB / s, 75.54MB / s].

[0144] The new threshold range is from 38.46MB / s to 75.54MB / s, and the system will automatically adapt to load changes based on new data. If the data flow exceeds this range, the system will identify it as an abnormal situation.

[0145] 3. Predictive warning based on pattern mining

[0146] By analyzing the patterns in historical monitoring data, possible abnormal trends can be discovered in advance and early warnings can be issued before the problem is fully exposed. The specific implementation process is as follows:

[0147] 1. Real-time data monitoring and pattern matching

[0148] After mining patterns in historical monitoring data, the system will predict potential anomalies during real-time monitoring by matching real-time data with historical patterns.

[0149] (1) Real-time data collection

[0150] The system continuously monitors real-time data on key metrics (such as data flow, CPU usage, etc.) and compares this data with known historical patterns.

[0151] (2) Pattern matching

[0152] The system analyzes the collected data in real time to determine whether it matches known historical patterns. If a match is successful, the system predicts potential anomalies.

[0153] For example, the system monitors that the current data traffic and CPU usage are gradually increasing, and network latency and response time are also increasing. If this trend matches the historical pattern, the system will think that an abnormality is about to occur and take action in advance.

[0154] Example data:

[0155] Assume that the system monitors the following data in Table 3 at 12:05:00.

[0156] Table 3

[0157] Timestamp Data flow CPU usage Memory usage Network latency Response time 12:05:00 85 88 82 140ms 1.7s

[0158] The system detects that data traffic and CPU usage are gradually increasing, and network delay and response time are also rising significantly, which matches the association rules and sequence pattern mining in Section 1. Therefore, the system predicts that network delay and response time may continue to increase, and identifies that anomalies may occur in the next few minutes.

[0159] (3) Issue an early warning

[0160] Once the system matches a potential abnormal pattern, it will issue an early warning based on the analysis results. Alert notifications can be made in a variety of ways, including email, SMS, system pop-up windows, etc. The content of the notification usually describes the abnormal situation in detail and provides possible recommended measures to ensure that operation and maintenance personnel can respond in a timely manner.

[0161] The trigger conditions for alarm notifications are usually based on pattern matching, for example:

[0162] When the data traffic exceeds 60MB / s and the CPU usage exceeds 75%, the system detects that it matches the association rule and triggers an alarm.

[0163] When data traffic and CPU usage gradually increase over several consecutive minutes, and network latency and response time increase significantly, the system detects a match with sequence pattern mining and triggers an alarm.

[0164] Example data:

[0165] Alarm notification content:

[0166] Time: 12:05:00;

[0167] Alarm level: high;

[0168] Alarm content:

[0169] The system detects abnormal patterns: data traffic and CPU usage are outside normal ranges;

[0170] Abnormal mode:

[0171] Matching historical association rules: data traffic > 60MB / s, CPU usage > 75%, which may lead to increased network delay and response time;

[0172] Matching historical sequence pattern mining: Data traffic and CPU usage continue to increase, and network performance is expected to deteriorate further in the next few minutes;

[0173] Suggested actions:

[0174] Please check the network load and consider reducing data traffic or allocating more computing resources as appropriate.

[0175] Based on the same inventive concept, the present invention also proposes an abnormal data prediction system based on dynamic threshold and pattern mining. The implementation of the system can refer to the implementation of the above method, and the repetitions will not be repeated. The term "module" used below can be a combination of software and / or hardware for designing a predetermined function. Although the system described in the following embodiments is preferably designed with software, the design of hardware, or a combination of software and hardware is also possible and conceived.

[0176] Figure 2 FIG. 1 is a schematic diagram of the structure of the abnormal data prediction system based on dynamic threshold and pattern mining of the present invention. Figure 2 As shown, the system includes:

[0177] The association rule and sequence pattern mining module 101 is used to identify the association relationship between multiple monitoring indicators under different data collection sources and extract frequently occurring sequence patterns by mining the association rules and sequence patterns of historical monitoring data.

[0178] Association rule mining is achieved through the following steps:

[0179] Find frequently occurring item sets by scanning historical monitoring data;

[0180] Based on frequently occurring item sets, association rules are generated and their support and confidence are calculated.

[0181] Sequential pattern mining is achieved through the following steps:

[0182] Divide historical monitoring data into multiple time windows in chronological order, and extract the monitoring indicator values ​​in the time windows;

[0183] Analyze the monitoring indicator values ​​in different time windows and generate candidate sequence patterns;

[0184] By scanning these candidate sequence patterns, the sequence patterns that frequently appear in multiple time windows are found;

[0185] Extract frequently occurring sequential patterns from time series.

[0186] The adaptive dynamic threshold module 102 is used to set the dynamic threshold range based on the mean and standard deviation of the historical monitoring data; and dynamically adjust the threshold range according to the real-time operating status and the change trend of the historical monitoring data.

[0187] The predictive warning module 103 is used to predict potential anomalies and issue warnings by matching real-time data with historical patterns.

[0188] It should be noted that although several modules of the abnormal data prediction system based on dynamic thresholds and pattern mining are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be concretized.

[0189] Based on the above invention concept, Figure 3 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, the aforementioned abnormal data prediction method based on dynamic thresholds and pattern mining is designed.

[0190] Based on the aforementioned inventive concept, the present invention further proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned abnormal data prediction method based on dynamic threshold and pattern mining.

[0191] The abnormal data prediction method and system based on dynamic threshold and pattern mining proposed in the present invention have the following highlights:

[0192] 1. By introducing an intelligent predictive early warning mechanism, a strategic transformation from traditional "post-event response" to "pre-event insight" has been achieved. The system can predict potential abnormal situations in advance based on historical data mining, issue warning notifications in advance, avoid the expansion of problems or system crashes, and ensure business continuity and high availability of the system.

[0193] 2. Using advanced adaptive dynamic threshold adjustment algorithms, the system can dynamically adjust the monitoring threshold according to the current operating load and environment. This flexible and accurate threshold adjustment mechanism can effectively reduce false positives and false negatives, ensuring efficient anomaly detection capabilities under different loads.

[0194] 3. By using cutting-edge technologies such as association rule mining and time series pattern recognition, the system can automatically extract complex abnormal patterns from massive historical data. This not only greatly expands the system's data insight capabilities, but also accurately captures the potential connections and trend changes between various monitoring indicators, builds a high-dimensional anomaly detection framework, and forms a more comprehensive early warning system.

[0195] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0196] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. An abnormal data prediction method based on dynamic threshold and pattern mining, characterized in that: The method includes: By mining the association rules and sequence patterns of historical monitoring data, we can identify the associations between multiple monitoring indicators under different data collection sources and extract frequently occurring sequence patterns. Set dynamic threshold ranges based on the mean and standard deviation of historical monitoring data; dynamically adjust threshold ranges based on real-time operating status and changing trends of historical monitoring data; By matching real-time data with historical patterns, potential anomalies can be predicted and early warnings can be issued.

2. The abnormal data prediction method based on dynamic threshold and pattern mining according to claim 1 is characterized in that: Association rule mining is achieved through the following steps: Find frequently occurring item sets by scanning historical monitoring data; Based on frequently occurring item sets, association rules are generated and their support and confidence are calculated.

3. The abnormal data prediction method based on dynamic threshold and pattern mining according to claim 1 is characterized in that: Sequential pattern mining is achieved through the following steps: Divide historical monitoring data into multiple time windows in chronological order, and extract the monitoring indicator values ​​in the time windows; Analyze the monitoring indicator values ​​in different time windows and generate candidate sequence patterns; By scanning these candidate sequence patterns, the sequence patterns that frequently appear in multiple time windows are found; Extract frequently occurring sequential patterns from time series.

4. An abnormal data prediction system based on dynamic threshold and pattern mining, characterized in that: The system includes: The association rule and sequence pattern mining module is used to identify the association relationship between multiple monitoring indicators under different data collection sources and extract frequently occurring sequence patterns through association rule and sequence pattern mining of historical monitoring data; Adaptive dynamic threshold module, used to set dynamic threshold range based on the mean and standard deviation of historical monitoring data; dynamically adjust threshold range according to real-time operation status and change trend of historical monitoring data; The predictive warning module is used to predict potential anomalies and issue warnings by matching real-time data with historical patterns.

5. The abnormal data prediction system based on dynamic threshold and pattern mining according to claim 4 is characterized in that: Association rule mining is achieved through the following steps: Find frequently occurring item sets by scanning historical monitoring data; Based on frequently occurring item sets, association rules are generated and their support and confidence are calculated.

6. The abnormal data prediction system based on dynamic threshold and pattern mining according to claim 4, characterized in that: Sequential pattern mining is achieved through the following steps: Divide historical monitoring data into multiple time windows in chronological order, and extract the monitoring indicator values ​​in the time windows; Analyze the monitoring indicator values ​​in different time windows and generate candidate sequence patterns; By scanning these candidate sequence patterns, the sequence patterns that frequently appear in multiple time windows are found; Extract frequently occurring sequential patterns from time series.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 3.

Images