Method and system for improving fault recall rate in intelligent anomaly detection scene
By collecting and preprocessing data from multi-source heterogeneous operation and maintenance data and selecting prediction models that are suitable for different scenarios, the problem of insufficient fault recall in intelligent anomaly detection is solved, and efficient fault detection and system stability improvement is achieved.
Patent Information
- Application Number
- CN202510752770.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-06
- Publication Date
- 2025-09-02
AI Technical Summary
The existing intelligent anomaly detection technology has shortcomings in fault recall, especially when facing complex and changeable data distribution and dynamic system environments, it is difficult to adapt to data fluctuations, resulting in a large number of real faults being missed, and insufficient model updates and adaptive adjustments.
By collecting data from multi-source heterogeneous operation and maintenance data, pre-processing and multi-stream merging, we select prediction models that are suitable for different scenarios, such as ARIMA, Holt-Winter and LSTM models, and automatically calculate thresholds in combination with statistical models, use classification algorithms to perform abnormal detection and normalization processing, and handle exceptions manually.
It significantly improves the fault recall rate, reduces missed detection, enhances the detection ability of complex data modes and subtle fault characteristics, improves the accuracy and reliability of detection, and enhances the safety and stability of the system.
Smart Images
Figure CN120579110A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent anomaly detection technology, and in particular to a method and system for improving the fault recall rate in intelligent anomaly detection scenarios. Background Art
[0002] In the field of intelligent operations and maintenance, intelligent anomaly detection plays a key role in ensuring stable system operation and preventing potential failures. However, existing intelligent anomaly detection technologies have significant shortcomings in fault recall. Traditional anomaly detection methods typically use fixed thresholds or simple statistical models to determine whether an anomaly exists. This approach often exhibits low recall rates when faced with complex and changing data distributions and dynamic system environments. Because fixed thresholds are difficult to adapt to data fluctuations, they can lead to a large number of real faults being missed.
[0003] Furthermore, existing anomaly detection systems have shortcomings in model updating and adaptive adjustment. When the system's operating state changes or new failure modes emerge, the model cannot learn and adapt in a timely manner, resulting in a gradual decline in the failure recall rate. Summary of the Invention
[0004] Purpose of the invention: To propose a method for improving the fault recall rate in intelligent anomaly detection scenarios, and to propose a system for implementing the above method, so as to adapt different models to different scenarios, improve the fault recall rate in intelligent anomaly detection scenarios, reduce the missed detection of potential faults, and thus solve the above problems existing in the prior art.
[0005] In a first aspect of the present invention, a method for improving the fault recall rate in an intelligent anomaly detection scenario is proposed, comprising the following steps: S1. Collecting original multi-source heterogeneous operation and maintenance data from different data sources, including facilities, operating systems, basic components, middleware, databases, and information systems; S2. Performing preprocessing operations on the acquired original multi-source heterogeneous operation and maintenance data, including multi-stream merging, data cleaning, and indicator sorting, and storing the preprocessed data; S3. Perform indicator prediction on the preprocessed data. Select different prediction models based on different scenarios (such as increasing business volume, stable business volume, and decreasing business volume). Compare the deviation between the actual value and the predicted value to determine whether the system has any anomalies. S4. Analyze the pre-processed data and automatically calculate the threshold of the predicted value of the indicator through the statistical model based on the volatility of the data itself; S5. Analyze the entire historical data to predict future business trends. Combined with current indicators and thresholds, predict the trends of future indicators and thresholds and adjust the sensitivity of anomaly detection. S6. Output the current abnormal situation and use a classification algorithm to normalize the current abnormal output; S7. After receiving the abnormal output, the current abnormality is processed. After the processing is completed, the abnormality detection module continues to analyze the processed data.
[0006] In a further embodiment of the first aspect, in step S1, data is collected from server facilities, network facilities, and storage facilities through the SNMP protocol and the IPMI protocol, and data is collected from the operating system and middleware through system commands and tools, including CPU usage, memory usage, disk I / O, process performance, and system service performance information, and logs of the information system are collected through log collection tools, including user behavior data and business-related data.
[0007] In a further embodiment of the first aspect, in step S2, the data preprocessing module merges multiple streams of the acquired data, merges the data using a message queue according to the type of data, sends data of the same type from multiple data sources to different topics, reads data from multiple topics through consumers and merges them; unifies timestamps and deduplicates duplicate data; and classifies the acquired data according to indicator type.
[0008] In a further embodiment of the first aspect, in step S3, different prediction models are selected according to preset scenarios: the ARIMA model is selected for a scenario with increasing business volume, the Holt-Winter model is selected for a scenario with flat business volume, and the LSTM model is selected for a scenario with decreasing business volume.
[0009] In a further embodiment of the first aspect, in step S4, the threshold of the predicted value of the indicator is automatically calculated through a statistical model, and the statistic Z is defined using a prediction formula. It is assumed that the current value is y and the predicted value is y′. When the predicted value deviates from the actual value, the absolute value of Z increases. The distribution value cluster of Z is estimated through a density estimation algorithm, and the 99.99 percentile value is selected as the final abnormality judgment threshold.
[0010] In a further embodiment of the first aspect, in step S5, a linear model is used to fit the business trend. When the business volume increases, the observation value continues to increase, and when the business volume is flat or decreasing, the observation value continues to decrease. The Huber-Regression algorithm is used to predict and analyze the observation values within the short-term window of the indicator to obtain the volatility of the indicator within the window, and the volatility is measured to predict the subsequent business trend.
[0011] In a further embodiment of the first aspect, in step S6, when an abnormal situation occurs, the indicators of the abnormal situation are analyzed to see whether there are periodic fluctuations or changes, and compared with business trends. If it does not conform to the normal trend or pattern, the classification algorithm is used to normalize the current abnormal output.
[0012] In a further embodiment of the first aspect, in step S7, the abnormality is handled manually, including blocking traffic, manually restarting relevant components, and manually repairing relevant data.
[0013] In a second aspect of the present invention, a system for improving the fault recall rate in intelligent anomaly detection scenarios is proposed. The system includes a performance indicator acquisition module, a data preprocessing module, an anomaly detection module, and an anomaly processing module.
[0014] The performance indicator collection module is used to collect original multi-source heterogeneous operation and maintenance data from different data sources, including but not limited to facilities, operating systems, basic components, middleware, databases, and information systems; The data preprocessing module merges multiple streams of acquired data, cleans the data, organizes indicators, and stores the preprocessed data; The data preprocessing module cleans, denoises and standardizes the acquired data to ensure data consistency; The anomaly detection module analyzes and detects the preprocessed data. The first step is to predict the indicators and select different prediction models according to different scenarios (such as business volume increase scenario, business volume stabilization scenario, and business volume decrease scenario). By comparing the deviation between the actual value and the predicted value, it is determined whether the system has an anomaly. The second step is to analyze the preprocessed data and automatically calculate the threshold of the predicted value of the indicator based on the volatility of the data itself through the statistical model. The third step is to analyze the entire historical data to predict subsequent business trends, and combine the current indicators and thresholds to predict the trend of subsequent indicators and thresholds, and adjust the sensitivity of anomaly detection. The fourth step is to output the current anomaly situation and use the classification algorithm to normalize the current anomaly output to ensure that the anomaly prediction module can converge quickly and ensure the prediction accuracy; After receiving the exception output, the exception handling module processes the current exception. After the processing is completed, the exception detection module will continue to analyze the processed data to ensure that the system returns to normal.
[0015] In a further embodiment of the second aspect, the performance indicator collection module is used to collect original multi-source heterogeneous operation and maintenance data from different data sources, including but not limited to facilities, operating systems, basic components, middleware, databases, and information systems, and label and store the relevant data according to different data characteristics.
[0016] Different data sources collect original multi-source heterogeneous operation and maintenance data, collect data on server facilities, network facilities, and storage facilities through SNMP and IPMI protocols, and collect data on operating systems and middleware through system commands and tools, including but not limited to CPU usage, memory usage, disk I / O, process performance, system service performance and other information. Log collection tools are used to collect logs of information systems, including user behavior data and business-related data.
[0017] In a further embodiment of the second aspect, the data preprocessing module performs multi-stream merging, data cleaning, and index sorting on the acquired data, and stores the preprocessed data.
[0018] The data preprocessing module merges multiple streams of acquired data and uses message queues to merge data based on the data type (such as facilities, operating systems, basic components, middleware, databases, and information systems). In the message queue, data from multiple data sources of the same type are sent to different topics. Consumers read data from multiple topics and merge them in the message queue. After the merger, the data is processed according to the key-value tag attributes of the message. For example, data from topic routers and switches are merged to form time series data of the network link.
[0019] The data preprocessing module cleans the acquired data, unifies timestamps, and removes duplicate data. If the collected data contains timestamps, different data sources may use different date and time formats, some in YYYY-MM-DD HH:MM:SS format, and some in Unix timestamps. The data preprocessing module must unify the timestamps into a single format. For text-based data without clear unique identifiers (such as operating system logs, facility logs, and middleware logs), the data preprocessing module uses a built-in text similarity algorithm to identify and remove similar data. The similarity threshold is typically set to 100%.
[0020] The data preprocessing module organizes the acquired data into indicators and classifies it by indicator type. For example, all performance-related data (such as time series data on CPU usage and historical memory usage) can be grouped into one category, while security-related data (such as firewall logs and intrusion detection system data) can be grouped into another. Data can also be classified based on its source (such as data from performance monitoring tools) or keywords (such as data containing the word "security" into the security category).
[0021] In a further embodiment of the second aspect, the anomaly detection module analyzes and detects preprocessed data. The first step is to predict indicators, select different prediction models based on different scenarios (such as business volume increase scenario, business volume flat scenario, and business volume decrease scenario), and determine whether the system has an anomaly by comparing the deviation between the actual value and the predicted value. The second step is to analyze the preprocessed data and automatically calculate the threshold of the predicted value of the indicator based on the volatility of the data itself using a statistical model (such as a linear regression model or a logistic regression model). The third step is to analyze the entire historical data to predict subsequent business trends, and combine the current indicators and thresholds to predict the trend of subsequent indicators and thresholds, and adjust the sensitivity of anomaly detection. The fourth step is to output the current anomaly situation and use a classification algorithm to normalize the current anomaly output to ensure that the anomaly prediction module can converge quickly and ensure prediction accuracy.
[0022] In a further embodiment of the second aspect, the anomaly detection module selects different prediction models based on preset scenarios, such as a business volume increase scenario, a business volume flat scenario, and a business volume decrease scenario. For example, the ARIMA model is selected for a business volume increase scenario, the HoIt-Winter model is selected for a business volume flat scenario, and the LSTM model is selected for a business volume decrease scenario. After selecting the model, the data from the last 5 days is input into the model. The key parameters of the model include the input dimension (input_dim), the number of hidden units (hidden_units), the number of layers (num_layers), and the output dimension (output_dim). The input dimension depends on the number of features of the input data. For example, if only the CPU usage indicator is considered, the input dimension is 1. Considering the performance indicator, the CPU, memory, and disk indicators need to be summarized, and the input dimension is 3. The number of hidden units is usually calculated using different values (such as 32, 64, 128, etc.), and the optimal number is finally determined based on the results. The number of layers is generally set to 1-3 layers to finally complete the prediction of the indicator.
[0023] In a further embodiment of the second aspect, the anomaly detection module automatically calculates a threshold for the predicted value of the indicator. Based on the volatility of the data itself, the threshold for the predicted value of the indicator is automatically calculated using a statistical model (such as a linear regression model or a logistic regression model). A prediction formula is used to define the statistic Z. Assuming the current value is y and the predicted value is y′, when the predicted value is very close to the actual value, the Z value is close to 0. The greater the deviation, the larger the absolute value of Z. Typically, the predicted value is very close to the actual value. That is, when the statistical model is used to calculate the indicator, when the mean and variance of the Z statistic calculated based on different time periods stabilize at a certain value, this value can be used as a reference for the threshold, ultimately forming a distribution cluster of Z values. The distribution cluster of Z values is estimated using a density estimation algorithm (such as the KDE algorithm), and the 99.99th percentile value is selected as the final anomaly judgment threshold.
[0024] In a further embodiment of the second aspect, the anomaly detection module analyzes the entire historical data set to predict future business trends. Combining current indicators and thresholds, it predicts the trends of future indicators and thresholds and adjusts the sensitivity of anomaly detection. Typically, business volume fluctuates periodically over time. If the prediction window is fixed within a recent finite time range and a linear model (such as a ridge regression model or a lasso model) is used to fit business trends, when business volume increases, the observed values increase over time. When business volume is flat or declining, the observed values decrease over time. The Huber-Regression algorithm is selected to predict and analyze the observed values within a short-term indicator window, deriving the volatility of the indicator within the window, measuring this volatility, and predicting future business trends. After the business trend prediction is completed, the anomaly detection sensitivity is appropriately lowered for some important business scenarios (such as those that affect user assets or user experience) through manual intervention.
[0025] In a further embodiment of the second aspect, the anomaly detection module uses a classification algorithm to normalize the current anomaly output, ensuring rapid convergence of the anomaly prediction module and ensuring prediction accuracy. When an anomaly occurs, it is necessary to analyze whether the anomaly indicator exhibits periodic fluctuations or changes, and compare it with business trends to determine whether it conforms to a normal trend or pattern. If the trend or pattern does not conform, the current anomaly output is normalized using a classification algorithm to ensure rapid convergence of the anomaly prediction module. For example, if certain anomaly indicators (such as CPU usage, memory usage, network bandwidth, etc.) do not conform to historical anomaly data (such as the rate of change or difference of indicators), in addition to selecting the current CPU usage as the detection indicator, the rate of change of CPU usage over the past five days can also be used as a weighted indicator.
[0026] In a further embodiment of the second aspect, upon receiving the exception output, the exception handling module processes the current exception. After processing is complete, the exception detection module continues to analyze the processed data to ensure that the system returns to normal. Exception processing typically involves manual methods, such as blocking traffic, manually restarting related components, or manually repairing related data.
[0027] In a further embodiment of the second aspect, in the system for improving the fault recall rate in the intelligent anomaly detection scenario, the data input and output formats are in XML format or JSON format, and jsonp cross-domain access is supported.
[0028] Compared with the existing technology, the present invention has the following beneficial effects: significantly improving the fault recall rate in intelligent anomaly detection scenarios, reducing the missed detection of potential faults, improving the security and stability of the system, enhancing the detection capability of complex data patterns and subtle fault characteristics, improving the accuracy and reliability of detection, and improving the overall performance and practical value of the intelligent anomaly detection system, providing strong guarantees for the safe production and efficient operation of enterprises. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 This is a flowchart of an intelligent anomaly detection scenario in an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In the following description, numerous specific details are provided to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without one or more of these details. In other instances, certain technical features well known in the art have not been described to avoid confusion with the present invention.
[0031] like Figure 1 The present invention provides a system for improving the fault recall rate in intelligent anomaly detection scenarios. The system consists of a performance indicator acquisition module, a data preprocessing module, an anomaly detection module, and an anomaly processing module.
[0032] The performance indicator collection module collects raw, multi-source, heterogeneous operational data from various data sources, including but not limited to facilities, operating systems, infrastructure components, middleware, databases, and information systems. The data preprocessing module merges multiple streams of acquired data, cleans the data, organizes indicators, and stores the preprocessed data. The anomaly detection module analyzes and detects the preprocessed data. The first step is to predict indicators, selecting different prediction models based on different scenarios (e.g., increasing business volume, flat business volume, and decreasing business volume). The module compares the deviation between actual and predicted values to determine whether the system has anomalies. The second step is to analyze the preprocessed data and, based on the data's inherent volatility, automatically calculate thresholds for the predicted indicator values using statistical models. The third step is to analyze the entire historical data set to predict future business trends. Combining current indicators with thresholds, the module predicts future trends for these indicators and thresholds, and adjusts the sensitivity of anomaly detection. The fourth step is to output the current abnormal situation and use the classification algorithm to normalize the current abnormal output to ensure that the abnormal prediction module can converge quickly and ensure the prediction accuracy; after receiving the abnormal output, the abnormal processing module processes the current abnormality. After the processing is completed, the abnormal detection module will continue to analyze the processed data to ensure that the system returns to normal. The present invention brings many significant beneficial effects, significantly improving the fault recall rate in the intelligent abnormality detection scenario, reducing the missed detection of potential faults, improving the security and stability of the system, enhancing the detection ability of complex data patterns and subtle fault characteristics, improving the accuracy and reliability of detection, and improving the overall performance and practical value of the intelligent abnormality detection system, providing a strong guarantee for the safe production and efficient operation of enterprises.
[0033] The anomaly detection module of the present invention selects different models based on pre-set scenarios: the ARIMA model is used in scenarios with increasing business volume, the HoIt-Winter model is used in scenarios with stable business volume, and the LSTM model is used in scenarios with decreasing business volume. In scenarios with increasing business volume, the data volume increases and the time window narrows. In these scenarios, the statistical characteristics of the data (such as the mean and variance) are less affected by the wide time window, resulting in a stable fluctuation range, and the mean and variance remain relatively stable over a long period of time. The ARIMA model primarily focuses on short-term forecasting. After data stabilization, the ARIMA model can infer short-term data trends based on the autoregressive and moving average relationships within the narrow time window. In scenarios with stable business volume, the data volume is relatively stable, the indicator trend changes to a limited extent, and the data exhibits seasonal characteristics. The HoIt-Winter model primarily focuses on medium-term forecasting, predicting indicator trends for the next month to a quarter. In scenarios with decreasing business volume, the data volume decreases rapidly, and the indicators exhibit dynamic changes. The LSTM model is suitable for processing long-series data and long-term forecasting, leveraging long-term patterns in historical data for prediction.
[0034] The data input and output formats in the present invention are XML format or JSON format, and support jsonp cross-domain access.
[0035] In the present invention, the data preprocessing module merges multiple streams of acquired data and uses a message queue to merge the data according to the type of data (such as facilities, operating systems, basic components, middleware, databases, and information systems). In the message queue, data from multiple data sources of the same type are sent to different topics. Data is read from multiple topics through consumers and merged in the message queue. After the merger, the merging operation is performed according to the key-value tag attributes of the message during the data processing process. For example, data from topic routers and switches are merged to form time series data of the network link.
[0036] The data preprocessing module in this invention cleans the acquired data, unifies timestamps, and removes duplicate data. For example, if the collected data contains timestamps, different data sources may use different date and time formats, some in YYYY-MM-DD HH:MM:SS format, and some in Unix timestamps. The data preprocessing module must unify the timestamps into a single format. For text-based data without clear unique identifiers (such as operating system logs, facility logs, and middleware logs), the data preprocessing module uses a built-in text similarity algorithm to identify and remove similar data. The similarity threshold is typically set to 100%.
[0037] The data preprocessing module of the present invention organizes the acquired data into indicators and classifies the data according to indicator type. For example, all performance-related data (such as time series data of CPU utilization and historical records of memory usage) can be classified into one category, while security-related data (such as firewall logs and intrusion detection system data) can be classified into another category. Categorization can also be performed based on data source (such as data from performance monitoring tools being classified as performance) or data keywords (such as data containing the word "security" being classified as security).
[0038] The anomaly detection module in the present invention predicts indicators. It selects different prediction models based on preset scenarios, such as those for increasing traffic, flat traffic, and decreasing traffic. For example, the ARIMA model is selected for increasing traffic, the HoIt-Winter model is selected for flat traffic, and the LSTM model is selected for decreasing traffic. After selecting a model, data from the last five days is input into the model. Key parameters of the model include the input dimension (input_dim), the number of hidden units (hidden_units), the number of layers (num_layers), and the output dimension (output_dim). The input dimension depends on the number of features in the input data. For example, if only CPU utilization is considered, the input dimension is 1. If performance indicators are considered, the CPU, memory, and disk indicators need to be summarized, and the input dimension is 3. The number of hidden units is typically calculated using different values (such as 32, 64, 128, etc.), and the optimal number is ultimately determined based on the results. The number of layers is generally set to 1-3, completing the indicator prediction.
[0039] The anomaly detection module in the present invention automatically calculates the threshold value of the predicted value of the indicator. Based on the volatility of the data itself, the threshold value of the predicted value of the indicator is automatically calculated through a statistical model (such as a linear regression model or a logistic regression model). , usually need to define the statistic Z, assuming the current value is y, the predicted value is When the predicted value is very close to the actual value, the Z value is close to 0. The greater the deviation, the larger the absolute value of Z. Typically, when the predicted value is very close to the actual value, that is, when the indicator is calculated using a statistical model, the mean and variance of the Z statistic calculated based on different time periods stabilize at a certain value. This value can be used as a reference for the threshold, ultimately forming a distribution cluster of Z values. This distribution cluster of Z values is estimated using a density estimation algorithm (such as the KDE algorithm), and the 99.99th percentile value is selected as the final anomaly judgment threshold.
[0040] The anomaly detection module in this invention analyzes the entire historical data set to predict future business trends. It then combines current indicators and thresholds to predict trends for subsequent indicators and thresholds, and adjusts the sensitivity of anomaly detection. Typically, business volume fluctuates periodically over time. If the prediction window is fixed within a recent finite time range and a linear model (such as a ridge regression model or a lasso model) is used to fit business trends, when business volume increases, the observed values increase over time. When business volume is flat or declining, the observed values decrease over time. The Huber-Regression algorithm is selected to predict and analyze the observed values within a short-term indicator window, deriving the volatility of the indicator within the window, measuring this volatility, and predicting future business trends. After the business trend prediction is completed, the anomaly detection sensitivity is appropriately lowered for some important business scenarios (such as those that affect user assets or user experience) through manual intervention.
[0041] In this invention, the anomaly detection module's current anomaly output is normalized to ensure rapid convergence of the anomaly prediction module and guarantee prediction accuracy. When an anomaly occurs, the anomaly indicator is analyzed for periodic fluctuations or changes, and compared with business trends to determine if it conforms to a normal trend or pattern. If not, a classification algorithm is used to normalize the current anomaly output to ensure rapid convergence of the anomaly prediction module. For example, if certain abnormal indicators (such as CPU usage, memory usage, and network bandwidth) do not conform to historical abnormal data (such as the rate of change or difference in indicators), in addition to selecting the current CPU usage as the detection indicator, the rate of change in CPU usage over the past five days can also be used as a weighted indicator.
[0042] After receiving the abnormal output, the exception handling module in the present invention handles the current abnormality, usually in a manual manner, such as blocking traffic, manually restarting related components, and manually repairing related data. After the processing is completed, the abnormality detection module will continue to analyze the processed data to ensure that the system returns to normal.
[0043] Based on the system disclosed in the above embodiments, this embodiment further discloses a method for improving the fault recall rate in intelligent anomaly detection scenarios, which includes the following steps: Step S1: Collect original multi-source heterogeneous operation and maintenance data from different data sources, including but not limited to facilities, operating systems, basic components, middleware, databases, and information systems. Specifically: Data on server facilities, network facilities, and storage facilities are collected through SNMP and IPMI protocols. Data on operating systems and middleware are collected through system commands and tools, including CPU usage, memory usage, disk I / O, process performance, and system service performance information. Log collection tools are used to collect logs from information systems, including user behavior data and business-related data.
[0044] Step S2: Merge multiple streams of the acquired data, clean the data, organize the indicators, and store the pre-processed data. Specifically: The data preprocessing module merges multiple streams of acquired data, uses message queues to merge data based on data type, sends data of the same type from multiple data sources to different topics, reads data from multiple topics through consumers and merges them; unifies timestamps and deduplicates duplicate data; and classifies acquired data according to indicator type.
[0045] Step S3: Predict indicators for the pre-processed data, select different prediction models based on different scenarios (such as business volume increase scenario, business volume stability scenario, business volume decrease scenario), and determine whether the system is abnormal by comparing the deviation between the actual value and the predicted value. Specifically: Select different prediction models based on the preset scenarios. For scenarios with increasing business volume, choose the ARIMA model; for scenarios with stable business volume, choose the Holt-Winter model; and for scenarios with decreasing business volume, choose the LSTM model.
[0046] Step S4: Analyze the pre-processed data and automatically calculate the threshold of the predicted value of the indicator through a statistical model (such as a linear regression model or a logistic regression model) based on the volatility of the data. Specifically: The threshold of the predicted value of the indicator is automatically calculated through the statistical model, and the statistic Z is defined using the prediction formula:
[0047] The current value is y, and the predicted value is y′. When the predicted value deviates from the actual value, the absolute value of Z increases. The distribution value cluster of Z is estimated through the density estimation algorithm, and the 99.99 percentile value is selected as the final anomaly judgment threshold.
[0048] Step S5: Analyze the entire historical data to predict future business trends, and combine the current indicators and thresholds to predict the trends of future indicators and thresholds, and adjust the sensitivity of anomaly detection. Specifically: A linear model is used to fit business trends. When the business volume increases, the observed value continues to increase. When the business volume is stable or decreasing, the observed value continues to decrease. The Huber-Regression algorithm is used to predict and analyze the observed values within the short-term window of the indicator, and the volatility of the indicator within the window is obtained. The volatility is measured and the subsequent business trends are predicted.
[0049] Step S6: Output the current abnormal situation and use the classification algorithm to normalize the current abnormal output to ensure that the abnormal prediction module can converge quickly and ensure the prediction accuracy. Specifically: When an abnormal situation occurs, analyze whether the indicators of the abnormal situation have periodic fluctuations or changes, and compare them with business trends. If they do not conform to normal trends or rules, use classification algorithms to normalize the current abnormal output.
[0050] Step S7: After receiving the exception output, the exception handling module processes the current exception. After processing is complete, the exception detection module continues to analyze the processed data to ensure that the system returns to normal. Exception handling is usually done manually, such as blocking traffic, manually restarting related components, and manually repairing related data.
[0051] The method and system for improving the fault recall rate in intelligent anomaly detection scenarios proposed in this embodiment significantly improve the fault recall rate in intelligent anomaly detection scenarios, reduce the missed detection of potential faults, improve the security and stability of the system, enhance the detection capabilities of complex data patterns and subtle fault characteristics, improve the accuracy and reliability of detection, and improve the overall performance and practical value of the intelligent anomaly detection system, providing strong guarantees for the safe production and efficient operation of enterprises.
[0052] The technical process of the method for improving the fault recall rate in the intelligent anomaly detection scenario disclosed in the above embodiment can be implemented in whole or in part through software, hardware, firmware or any other combination.
[0053] When implemented in hardware, the aforementioned embodiments can be run on an electronic device by compiling all or part of the operating logic and computational processes into software. The electronic device includes a processor, memory, a communication interface, and a communication bus. The processor, memory, and communication interface communicate with each other via the communication bus. The memory stores at least one executable instruction that causes the processor to execute the technical process of the method for improving fault recall rates in intelligent anomaly detection scenarios disclosed in the aforementioned embodiments.
[0054] When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. If the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be embodied in the form of a software product, which is essentially or contributes to the relevant technology. The software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk. In this way, the embodiments of the present application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.
[0055] In the embodiments disclosed herein, computer storage media may be tangible media that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. Computer storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media may include electrical connections based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0056] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0057] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.
Claims
1. A method for improving the fault recall rate in intelligent anomaly detection scenarios, characterized in that: The steps include: S1. Collecting original multi-source heterogeneous operation and maintenance data from different data sources, including facilities, operating systems, basic components, middleware, databases, and information systems; S2. Performing preprocessing operations on the acquired original multi-source heterogeneous operation and maintenance data, including multi-stream merging, data cleaning, and indicator sorting, and storing the preprocessed data; S3. Predict indicators for the preprocessed data, select different prediction models based on different scenarios, and determine whether the system is abnormal by comparing the deviation between the actual value and the predicted value; S4. Analyze the pre-processed data and automatically calculate the threshold of the predicted value of the indicator through the statistical model based on the volatility of the data itself; S5. Analyze the entire historical data to predict future business trends. Combined with current indicators and thresholds, predict the trends of future indicators and thresholds and adjust the sensitivity of anomaly detection. S6. Output the current abnormal situation and use a classification algorithm to normalize the current abnormal output; S7. After receiving the abnormal output, process the current abnormality and continue to analyze the processed data.
2. The method for improving the fault recall rate in intelligent anomaly detection scenarios according to claim 1 is characterized in that: In step S1, data on server facilities, network facilities, and storage facilities are collected through the SNMP protocol and the IPMI protocol, and data on the operating system and middleware are collected through system commands and tools, including CPU usage, memory usage, disk I / O, process performance, and system service performance information. Logs of the information system are collected through log collection tools, including user behavior data and business-related data.
3. The method for improving the fault recall rate in intelligent anomaly detection scenarios according to claim 1 is characterized in that: In step S2, the data preprocessing module merges multiple streams of acquired data, uses message queues to merge data based on data type, sends data of the same type from multiple data sources to different topics, and uses consumers to read data from multiple topics and merge them. Unify timestamps and deduplicate data; classify acquired data by indicator type.
4. The method for improving the fault recall rate in intelligent anomaly detection scenarios according to claim 1 is characterized in that: In step S3, different prediction models are selected according to preset scenarios. The ARIMA model is selected for a scenario with increasing business volume, the Holt-Winter model is selected for a scenario with stable business volume, and the LSTM model is selected for a scenario with decreasing business volume.
5. The method for improving the fault recall rate in intelligent anomaly detection scenarios according to claim 1 is characterized in that: In step S4, the threshold of the predicted value of the indicator is automatically calculated using the statistical model, and the statistic Z is defined using the prediction formula: The current value is y, and the predicted value is y′. When the predicted value deviates from the actual value, the absolute value of Z increases. The distribution value cluster of Z is estimated through the density estimation algorithm, and the 99.99 percentile value is selected as the final anomaly judgment threshold.
6. The method for improving the fault recall rate in intelligent anomaly detection scenarios according to claim 1, characterized in that: In step S5, a linear model is used to fit the business trend. When the business volume increases, the observation value continues to increase. When the business volume is flat or decreasing, the observation value continues to decrease. The Huber-Regression algorithm is used to predict and analyze the observation values within the short-term window of the indicator to obtain the volatility of the indicator within the window, and the volatility is measured to predict the subsequent business trend.
7. The method for improving the fault recall rate in intelligent anomaly detection scenarios according to claim 1 is characterized in that: In step S6, when an abnormal situation occurs, the indicators of the abnormal situation are analyzed to see whether there are periodic fluctuations or changes, and compared with the business trends. If they do not conform to the normal trend or rules, the classification algorithm is used to normalize the current abnormal output.
8. The method for improving the fault recall rate in intelligent anomaly detection scenarios according to claim 1, characterized in that: In step S7, the abnormality is handled manually, including blocking traffic, manually restarting related components, and manually repairing related data.
9. A system for improving the fault recall rate in intelligent anomaly detection scenarios, characterized in that: include: Performance indicator collection module, used to collect original multi-source heterogeneous operation and maintenance data from different data sources; The data preprocessing module is used to merge multiple streams of acquired data, perform data cleaning, organize indicators, and store the preprocessed data; Anomaly detection module, used to analyze and detect pre-processed data, including indicator prediction, automatic threshold calculation, business trend prediction and anomaly output; The exception handling module is used to handle the current exception. After the processing is completed, the exception detection module will continue to analyze the processed data to ensure that the system returns to normal.
10. The system for improving the fault recall rate in intelligent anomaly detection scenarios according to claim 9, characterized in that: The data input and output formats are XML or JSON, and support JSONP cross-domain access.