A device abnormal data detection method, device, equipment and storage medium

Abstract

The present application discloses a device abnormal data detection method, apparatus, device, and storage medium, relating to the field of computer technology, including: extracting information from historical time series data of a target device within a preset time period before the current moment according to different time scales to obtain multiple time scale series data; calculating the abnormality score of each time scale based on a preset sliding window size and the time scale series data to obtain multiple first abnormality scores; calculating the sensitivity of each time scale based on the first abnormality score and a preset score threshold, and calculating the abnormal data detection threshold based on the maximum value of the sensitivity; calculating the abnormality score of the target device's to-be-detected data at the current moment based on the preset sliding window size to obtain the current abnormality score; and determining whether the to-be-detected data is abnormal data based on the comparison result between the current abnormality score and the abnormal data detection threshold. The present application can improve the detection efficiency of abnormal data and meet the needs of real-time processing of large amounts of data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for detecting abnormal data in a device. Background Art

[0002] With the rapid development of data collection technology, the amount of time series data available in fields such as the Industrial Internet of Things, smart manufacturing, and equipment health management is also increasing. This data reflects the operating status of equipment. However, how to accurately identify anomalies or outliers in this massive amount of data and provide timely warnings of potential failures remains a pressing problem.

[0003] Traditional methods for detecting anomalies in massive amounts of data typically analyze data at a single time scale. This can lead to certain types of anomalies being overlooked. For example, some anomalies may not be noticeable on an hourly scale but may be highly significant on a daily or weekly scale. Furthermore, different devices may have different characteristics, making them suitable for anomaly detection at different time scales. Therefore, a method that can adaptively select the optimal detection scale is crucial.

[0004] However, the current mainstream methods for detecting abnormal data of devices, such as statistical anomaly detection methods and machine learning-based anomaly detection methods, also have some defects. For example, the Moving Average method and Standard Deviation method in statistical anomaly detection methods identify anomalies by calculating the deviation between a data point and the average value of its surrounding data, but usually only analyze on a single time scale (i.e., the time scale is fixed), so anomalies on other scales may be ignored. Moreover, the fixed threshold used by this method may not adapt to the dynamic changes of data, which may lead to false positives or false negatives. In addition, it is impossible to automatically adjust the detection strategy according to the characteristics of different devices, so it lacks adaptive capabilities. For example, the Isolation Forest method in machine learning-based anomaly detection methods This method identifies anomalies by constructing a decision tree. Specifically, it constructs a decision tree by randomly selecting features and split points, and repeats this step multiple times to build a decision tree forest. It then calculates the anomaly score of each data point and marks points with scores above the threshold as anomalies based on the average path length in the decision tree and a preset threshold. However, this method has high computational complexity and may not meet the requirements of real-time processing for large-scale real-time data. It also requires a large amount of training data and may perform poorly when data is scarce or a new system is deployed. In addition, it lacks interpretability and is difficult to explain why a point is judged to be anomaly, which is not conducive to subsequent fault diagnosis. It also fails to effectively utilize the characteristics of time series and ignores the temporal information and multi-scale characteristics of the data. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a device abnormal data detection method, apparatus, device, and storage medium that can capture abnormalities of different types and durations, improve the detection efficiency of abnormal data, and meet the needs of real-time processing of large amounts of data. The specific solution is as follows:

[0006] In a first aspect, the present application discloses a method for detecting abnormal data of a device, comprising:

[0007] According to different time scales, the historical time series data of the target device within a preset time period before the current moment is extracted to obtain multiple time scale series data;

[0008] Calculating anomaly scores for each time scale based on a preset sliding window size and the time scale sequence data to obtain a plurality of first anomaly scores;

[0009] Calculating the sensitivity of each time scale based on the first anomaly score and a preset score threshold, and calculating the abnormal data detection threshold based on the maximum value of the sensitivities;

[0010] Calculating, based on the preset sliding window size, an anomaly score of the to-be-detected data of the target device at the current moment to obtain a current anomaly score;

[0011] Whether the data to be detected is abnormal data is determined according to a comparison result between the current abnormality score and the abnormal data detection threshold.

[0012] Optionally, the anomaly score of each time scale is calculated based on the preset sliding window size and the time scale sequence data to obtain multiple first anomaly scores, including:

[0013] Calculating the moving average of each time scale based on a preset sliding window size and the time scale series data, and calculating the moving standard deviation of each time scale based on the moving average;

[0014] An anomaly score of each time scale is calculated based on the time scale series data, the moving standard deviation, and the moving average to obtain a plurality of first anomaly scores.

[0015] Optionally, the calculating of the moving average of each time scale based on the preset sliding window size and the time scale series data includes:

[0016] respectively calculating the sum of the plurality of data in each of the time-scale series data to obtain a first sum, and counting the number of the plurality of data in the time-scale series data to obtain a first statistical result;

[0017] The ratio of the first sum value to the first statistical result is calculated to obtain a first ratio, and the moving average of each time scale is calculated based on a preset sliding window size and the first ratio using a moving average method.

[0018] Optionally, the calculating of the anomaly score of each time scale based on the time scale series data, the moving standard deviation, and the moving average to obtain multiple first anomaly scores includes:

[0019] Calculating the difference between the first ratio and the moving average corresponding to each time scale to obtain a target difference;

[0020] The absolute value of the target difference is calculated, and the ratio of the absolute value to the moving standard deviation is calculated to obtain a first anomaly score corresponding to each time scale.

[0021] Optionally, calculating the sensitivity of each time scale based on the first anomaly score and a preset score threshold includes:

[0022] Obtaining scores greater than a preset score threshold from among the plurality of first anomaly scores corresponding to each time scale within the preset duration to obtain a second anomaly score, and counting the number of the second anomaly scores corresponding to each time scale to obtain a second statistical result;

[0023] Calculating a sum of multiple second anomaly scores corresponding to each time scale to obtain a second sum;

[0024] The ratio of the second sum value to the second statistical result is calculated to obtain the sensitivity of the corresponding time scale.

[0025] Optionally, calculating the abnormal data detection threshold based on the maximum value of the sensitivities includes:

[0026] Determine a maximum value among the multiple sensitivities corresponding to multiple time scales, and calculate the mean and standard deviation of the first anomaly score corresponding to the maximum value within the preset time length to obtain a target mean and a target standard deviation;

[0027] determining a target time scale corresponding to the maximum value, and obtaining a preset adjustable parameter corresponding to the target time scale;

[0028] The product of the target standard deviation and a preset adjustable parameter is calculated, and the sum of the product and the target mean is calculated to obtain an abnormal data detection threshold.

[0029] Optionally, determining whether the data to be detected is abnormal data according to a comparison result between the current abnormality score and the abnormal data detection threshold includes:

[0030] If the current anomaly score is greater than the abnormal data detection threshold, the data to be detected is determined to be abnormal data;

[0031] If the current anomaly score is not greater than the abnormal data detection threshold, the data to be detected is determined to be normal data.

[0032] Optionally, after determining whether the data to be detected is abnormal data according to the comparison result between the current abnormality score and the abnormal data detection threshold, the method further includes:

[0033] If the data to be detected is abnormal data, the number of abnormal data determined to be abnormal data within a preset time is counted to obtain a first number;

[0034] Obtaining the number of false alarms in the abnormal data determined within the preset time to obtain a second number;

[0035] Calculating a ratio of the second number to the first number to obtain a second ratio, and determining whether the second ratio is greater than a preset false alarm threshold;

[0036] If the second ratio is greater than the preset false alarm threshold, the value of the preset adjustable parameter is adjusted down, and the step of extracting information from the historical time series data of the target device within the preset time length before the current moment according to different time scales is triggered to obtain multiple time scale series data.

[0037] Optionally, the information is extracted from the historical time series data of the target device within a preset time period before the current moment according to different time scales to obtain multiple time scale series data, including:

[0038] When a preset time period is reached, historical operation data and / or historical status data of the components to be monitored in the server within a preset time period before the current moment are obtained to obtain historical time series data; the server is a server located in a preset data center;

[0039] Information is extracted from the historical time series data according to minute time scales, hour time scales, day time scales, and week time scales to obtain time scale series data including minute level, hour level, day level, and week level.

[0040] Optionally, the acquiring of historical operation data and / or historical status data of the component to be monitored in the server within a preset time period before the current moment to obtain historical time series data includes:

[0041] Get the CPU usage of the server recorded according to the preset collection period within the preset time period before the current time to obtain historical time series data.

[0042] Optionally, after determining whether the data to be detected is abnormal data according to the comparison result between the current abnormality score and the abnormal data detection threshold, the method further includes:

[0043] If the data to be detected is abnormal data, determining whether the data to be detected is greater than a preset usage rate threshold;

[0044] If the data to be detected is greater than the preset usage rate threshold, a load abnormality alarm is triggered.

[0045] Optionally, after determining whether the data to be detected is abnormal data according to the comparison result between the current abnormality score and the abnormal data detection threshold, the method further includes:

[0046] If the data to be detected is abnormal data, alarm information including the abnormal data, the abnormal data detection threshold and the current abnormality score is generated, so as to process the situation causing the abnormal data.

[0047] In a second aspect, the present application discloses a computer program product, including a computer program / instruction, which implements the aforementioned device abnormal data detection method when executed by a processor.

[0048] In a third aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the aforementioned device abnormal data detection method is implemented.

[0049] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned device abnormal data detection method is implemented.

[0050] It can be seen that the present application first extracts information from the historical time series data of the target device within a preset time length before the current moment according to different time scales to obtain multiple time scale series data, and then calculates the anomaly score of each time scale based on the preset sliding window size and the time scale series data to obtain multiple first anomaly scores, and then calculates the sensitivity of each time scale based on the first anomaly score and the preset score threshold, and calculates the anomaly data detection threshold based on the maximum value of the sensitivities, and then calculates the anomaly score of the data to be detected of the target device at the current moment based on the preset sliding window size to obtain the current anomaly score, and determines whether the data to be detected is abnormal data based on the comparison result between the current anomaly score and the anomaly data detection threshold. The present application extracts information from the historical time series data of the target device within a preset time period before the current moment according to different time scales, thereby obtaining sequence data of multiple different time scales. By converting a single time series data into sequence data of multiple different time scales, anomalies of different types and durations can be captured, such as short-term burst loads and long-term performance degradation trends; further, the anomaly score of each time scale is calculated based on the sequence data corresponding to each time scale and the preset sliding window size, and the sensitivity of each time scale is calculated based on each anomaly score. Finally, the anomaly data detection threshold for anomaly data detection is calculated based on the maximum value of the multiple sensitivities. Compared with traditional statistical and machine learning-based anomaly detection methods, this solution does not involve a large number of training processes and statistical processes, thereby improving the detection efficiency of anomaly data and meeting the needs of real-time processing of large amounts of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0052] Figure 1 This is a flow chart of a device abnormal data detection method disclosed in this application;

[0053] Figure 2 This is a flow chart of a specific device abnormal data detection method disclosed in this application;

[0054] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0056] The present application discloses a method for detecting abnormal data of a device. Figure 1 As shown, the method includes:

[0057] Step S11: extracting information from the historical time series data of the target device within a preset time period before the current moment according to different time scales to obtain multiple time scale series data.

[0058] In this embodiment, when it is necessary to perform abnormal data detection on the target device, the historical time series data of the target device within a preset time period before the current moment (such as within the past week) is first obtained, and then information is extracted from the above historical time series data according to different time scales (such as minute time scale, hour time scale, day time scale, week time scale, month time scale, etc.), thereby obtaining multiple time scale sequence data; wherein, the target device includes but is not limited to servers, computers and other devices, and the historical time series data can specifically be the historical time series data of the components to be monitored in the target device, such as the CPU (Central Processing Unit) usage rate, CPU temperature, memory usage rate, fan temperature or fan speed data recorded every 5 minutes in the server. By extracting sequence data of multiple time scales from the historical time series data, abnormal patterns of different types and durations can be captured.

[0059] For example, obtain the fan temperature series data recorded by the server system according to the preset monitoring cycle (every 5 minutes) in the past week, and then extract information from the above fan temperature series data according to the hourly time scale, daily time scale, and weekly time scale to obtain multiple time scale series data; among them, the time scale series data corresponding to the hourly time scale is the fan temperature data extracted with hours as the information extraction unit. Since the preset monitoring cycle is to collect data every 5 minutes, there are 12 data points extracted in 1 hour, and the average of these 12 data can be calculated as a valid data point corresponding to the hourly time scale; similarly, the data points extracted in 1 day are 12*24, and the average of these 12*24 data can be calculated to obtain a valid data point corresponding to the daily time scale; similarly, the data points extracted in 1 week are 12*24*7, and the average of these 12*24*7 data can be calculated to obtain a valid data point corresponding to the weekly time scale.

[0060] Step S12: Calculate the anomaly score of each time scale based on the preset sliding window size and the time scale sequence data to obtain a plurality of first anomaly scores.

[0061] In this embodiment, after extracting information from the historical time series data of the target device within a preset time period before the current moment according to different time scales to obtain multiple time scale series data, the anomaly score of each time scale can be further calculated based on the preset sliding window size k and the above-mentioned time scale series data to obtain multiple corresponding first anomaly scores.

[0062] In a specific embodiment, the calculation of the anomaly score of each time scale based on the preset sliding window size and the time scale sequence data to obtain multiple first anomaly scores may include: calculating the moving average of each time scale based on the preset sliding window size and the time scale sequence data, and calculating the moving standard deviation of each time scale based on the moving average; calculating the anomaly score of each time scale based on the time scale sequence data, the moving standard deviation, and the moving average to obtain multiple first anomaly scores. In this embodiment, the moving average of each time scale can be calculated based on a preset sliding window size k and the time scale sequence data, and then the moving standard deviation of each time scale can be calculated based on the above-mentioned moving average, and then the anomaly score of each time scale can be calculated based on the above-mentioned time scale sequence data, the above-mentioned moving standard deviation, and the above-mentioned moving average corresponding to each time scale, thereby obtaining the anomaly score of all time scales.

[0063] Specifically, the calculation of the moving average of each time scale based on the preset sliding window size and the time scale series data may include: calculating the sum of multiple data in each time scale series data to obtain a first sum, and counting the number of multiple data in the time scale series data to obtain a first statistical result; calculating the ratio of the first sum to the first statistical result to obtain a first ratio, and using the moving average method to calculate the moving average of each time scale based on the preset sliding window size and the first ratio. For example, when the time scale series data is recorded every 5 minutes, information extraction can be performed on the time scale series data according to the minute time scale, hour time scale, day time scale, and week time scale to obtain multiple time scale series data; wherein the minute time scale is the time series data originally recorded by the system, recorded as X_m(t) for t in minute; then, based on X_m(t), the sum of multiple data in the time scale series data corresponding to the hourly time scale, daily time scale, and weekly time scale (i.e., the above-mentioned valid data points) is calculated respectively, and the number of multiple data in the time scale series data is counted to obtain a statistical result, and then the ratio of the sum to the statistical result is calculated; wherein, the ratio calculation formula corresponding to the hourly time scale is X_h(t) = Σ(X_m(t)for t in hour) / 12, the ratio calculation formula corresponding to the day time scale is X_d(t) = Σ(X_h(t) for t inday) / 24, and the ratio calculation formula corresponding to the weekly time scale is X_w(t) = Σ(X_d(t) for t in week) / 7. Furthermore, a moving average method is used to calculate the moving average of each time scale based on a preset sliding window size k (e.g., k is 24 hours) and the above ratio. Specifically, the time scale can be represented as s in {m, h, d, w}, and then the moving average of each time scale is calculated as MA_s(t) = Σ(X_s(i) for i in [tk, t+k]) / (2k+1). Based on the moving average MA_s(t), the moving standard deviation of each time scale is calculated as SD_s(t) = sqrt(Σ((X_s(i) - MA_s(t))^2 for i in [tk, t+k]) / (2k+1)).

[0064] Specifically, calculating anomaly scores for each time scale based on the time scale series data, the moving standard deviation, and the moving average to obtain multiple first anomaly scores may include: calculating the difference between the first ratio corresponding to each time scale and the moving average to obtain a target difference; calculating the absolute value of the target difference, and calculating the ratio of the absolute value to the moving standard deviation to obtain the first anomaly score corresponding to each time scale. That is, calculating the difference between the first ratio corresponding to each time scale and the moving average, calculating the absolute value of the difference, and then calculating the ratio of the absolute value to the moving standard deviation to obtain the first anomaly score corresponding to each time scale. Specifically, the calculation formula for the first anomaly score Score_s(t) is: Score_s(t) = |X_s(t) - MA_s(t)| / SD_s(t).

[0065] Step S13: Calculate the sensitivity of each time scale based on the first anomaly score and a preset score threshold, and calculate the abnormal data detection threshold based on the maximum value of the sensitivities.

[0066] In this embodiment, after calculating the anomaly scores of each time scale based on the preset sliding window size and the time scale series data, the sensitivity of each time scale can be calculated based on the above-mentioned first anomaly score and the preset score threshold, and then the maximum value of all sensitivities is determined, and the threshold for abnormal data detection is calculated based on the maximum value.

[0067] It should be pointed out that in order to improve the accuracy of abnormal data detection, before performing device abnormal data detection, different adjustable parameters can be set for different time scales using the grid search method in advance, so as to obtain the preset adjustable parameters corresponding to each time scale.

[0068] Specifically, the calculation of the sensitivity of each time scale based on the first anomaly score and the preset score threshold, and the calculation of the abnormal data detection threshold based on the maximum value of the sensitivities may include: obtaining the scores greater than the preset score threshold from the multiple first anomaly scores corresponding to each time scale within the preset time length to obtain a second anomaly score, and counting the number of the second anomaly scores corresponding to each time scale to obtain a second statistical result; calculating the sum of the multiple second anomaly scores corresponding to each time scale to obtain a second sum; calculating the ratio of the second sum to the second statistical result to obtain the sensitivity of the corresponding time scale. In this embodiment, the scores of multiple first anomaly scores Score_s(t) corresponding to each time scale within a preset time period (e.g., within a week) that are greater than a preset score threshold are first obtained to obtain a second anomaly score. Then, the number of all second anomaly scores corresponding to each time scale is counted to obtain a second statistical result T. Next, the sum of the multiple second anomaly scores corresponding to each time scale is calculated to obtain a second sum, and the ratio of the second sum to the second statistical result T is calculated to obtain the sensitivity Sensitivity_s of the corresponding time scale. The specific calculation formula is: Sensitivity_s = Σ(Score_s(t) > threshold) / T.

[0069] Furthermore, the calculation of the abnormal data detection threshold based on the maximum value of the sensitivity may specifically include: determining the maximum value of multiple sensitivities corresponding to multiple time scales, and calculating the mean and standard deviation of the first abnormality score corresponding to the maximum value within the preset time length to obtain a target mean and a target standard deviation; determining the target time scale corresponding to the maximum value, and obtaining a preset adjustable parameter corresponding to the target time scale; calculating the product of the target standard deviation and the preset adjustable parameter, and calculating the sum of the product and the target mean to obtain the abnormal data detection threshold. In this embodiment, the maximum value Best_scale among all sensitivities corresponding to multiple time scales can be first determined. The specific calculation formula is: Best_scale = argmax(Sensitivity_s for s in {m, h, d, w}). Then, the mean and standard deviation of the first anomaly score Score_s(t) corresponding to the maximum value Best_scale within a preset time length (such as within one week) are calculated to obtain the target mean μ and target standard deviation σ. Finally, the target time scale corresponding to the maximum value Best_scale is determined, and the preset adjustable parameter α corresponding to the target time scale is obtained. Then, the product of the target standard deviation σ and the preset adjustable parameter α is calculated, and then the sum of the product and the target mean μ is calculated to obtain the abnormal data detection threshold Threshold_dynamic(t). The specific calculation formula is: Threshold_dynamic(t) = μ + α * σ.

[0070] It should be pointed out that the preset score threshold (threshold) is obtained by optimizing the initial score threshold threshold_initial, and the setting of the initial score threshold threshold_initial can be determined by a variety of methods. For example, based on the experience of the system administrator or domain expert (i.e., domain knowledge), a reasonable initial score threshold is set for each time scale; for example, time series data before the historical moment is collected, and an initial threshold is set by analyzing this data; for example, the initial score threshold is directly set to a higher value to avoid too many false alarms, and then the value is gradually adjusted as the system runs; or time series data before a historical moment is collected, and then the mean μ and standard deviation σ are calculated using this data, and then the initial score threshold is set. The specific calculation formula is: threshold_initial = μ + g * σ. In a specific implementation, g=2 or g=3, and this value depends on the required sensitivity_s.

[0071] It should also be noted that the preset adjustable parameter α is a tunable parameter used to control the impact of anomaly scores on the final result during the sensitivity calculation. This parameter value can be determined in a variety of ways. For example, based on experience with similar systems, an initial value, such as 0.1 or 0.2, can be set. Alternatively, it can be set based on the system's sensitivity to anomalies; for systems requiring greater sensitivity, α can be set larger. Alternatively, it can be set using a preset dynamic adjustment mechanism that allows α to dynamically adjust based on system performance, gradually reducing its value if the false alarm rate is too high. For example, during system initialization, a small amount of data, such as historical time series data from the past day, can be used to find an optimal α value through grid search. Furthermore, to adapt to the characteristics of different time scales, different α values ​​can be set for different time scales.

[0072] Step S14: Based on the preset sliding window size, the anomaly score of the to-be-detected data of the target device at the current moment is calculated to obtain a current anomaly score.

[0073] In this embodiment, after obtaining the abnormal data detection threshold, the target device's current data to be detected, such as fan temperature data and CPU usage, is obtained. Then, based on the preset sliding window size k, an anomaly score for the current data to be detected is calculated to obtain a current anomaly score. It should be noted that the calculation process for the current anomaly score is identical to the method for calculating the anomaly score in step S12 and will not be further described here.

[0074] Step S15: determining whether the data to be detected is abnormal data based on a comparison result between the current abnormality score and the abnormal data detection threshold.

[0075] In this embodiment, after the anomaly score of the current data to be detected is calculated, it can be directly determined whether the current data to be detected is abnormal data based on a comparison result between the current anomaly score and the abnormal data detection threshold.

[0076] Specifically, determining whether the data to be detected is abnormal data according to the comparison result between the current abnormal score and the abnormal data detection threshold may include: if the current abnormal score is greater than the abnormal data detection threshold, determining that the data to be detected is abnormal data; if the current abnormal score is not greater than the abnormal data detection threshold, determining that the data to be detected is normal data. That is, it is judged whether the abnormal score of the data to be detected is greater than the abnormal data detection threshold. If it is greater (i.e., Score_Best_scale(t) > Threshold_dynamic(t)), it is directly determined that the current data to be detected is abnormal data; if it is not less (i.e., Score_Best_scale(t) < Threshold_dynamic(t) or Score_Best_scale(t) = Threshold_dynamic(t)), it is determined that the current data to be detected is normal data.

[0077] Further, after determining whether the data to be detected is abnormal data according to the comparison result between the current abnormal score and the abnormal data detection threshold, it may further include: if the data to be detected is abnormal data, counting the number of data determined to be the abnormal data within a preset time to obtain a first number; obtaining the number of false alarms among the data determined to be the abnormal data within the preset time to obtain a second number; calculating the ratio of the second number to the first number to obtain a second ratio, and judging whether the second ratio is greater than a preset false alarm threshold; if the second ratio is greater than the preset false alarm threshold, adjusting the value of the preset adjustable parameter to be smaller, and triggering the step of extracting information from the historical time series data of the target device within a preset duration before the current moment according to different time scales to obtain multiple time scale series data. That is, after determining that the current data to be detected is abnormal data, counting the number of data determined to be the abnormal data within a preset time to obtain a first number, then obtaining the number of false alarms among the data determined to be abnormal data within the preset time to obtain a second number, and then calculating the ratio of the second number to the first number to obtain a second ratio. If the second ratio is greater than the preset false alarm threshold, adjusting the value of the preset adjustable parameter α to be smaller, and directly performing a new abnormal data check to calculate a new abnormal data detection threshold, and detecting the current data to be detected based on the new threshold. By continuously adjusting the preset adjustable parameter α that causes the false alarm rate to increase during the real-time detection process, it can adapt to the characteristics of different time scales.

[0078] Furthermore, after determining whether the data to be detected is abnormal data based on the comparison result between the current abnormality score and the abnormal data detection threshold, the method may further include: if the data to be detected is abnormal data, generating an alarm message including the abnormal data, the abnormal data detection threshold, and the current abnormality score, so as to address the situation causing the abnormal data. In this embodiment, to achieve interpretability of abnormal data detection, each step of the operation can be annotated with a clear physical meaning to facilitate the user's understanding and interpretation of the detection results. After determining that the current data to be detected is abnormal data, an alarm message including the abnormal data, the abnormal data detection threshold, and the current abnormality score can be generated, so that the user can be informed of the specific cause of the abnormality and can promptly address the situation causing the abnormal data.

[0079] It can be seen that the embodiment of the present application first extracts information from the historical time series data of the target device within a preset time length before the current moment according to different time scales to obtain multiple time scale series data, and then calculates the anomaly score of each time scale based on the preset sliding window size and the time scale series data to obtain multiple first anomaly scores, and then calculates the sensitivity of each time scale based on the first anomaly score and the preset score threshold, and calculates the anomaly data detection threshold based on the maximum value of the sensitivities, and then calculates the anomaly score of the data to be detected of the target device at the current moment based on the preset sliding window size to obtain the current anomaly score, and determines whether the data to be detected is abnormal data based on the comparison result between the current anomaly score and the anomaly data detection threshold. The embodiment of the present application extracts information from the historical time series data of the target device within a preset time period before the current moment according to different time scales, thereby obtaining sequence data of multiple different time scales. By converting a single time series data into sequence data of multiple different time scales, anomalies of different types and durations can be captured, such as short-term burst loads and long-term performance degradation trends. Furthermore, the anomaly score of each time scale is calculated based on the sequence data corresponding to each time scale and the preset sliding window size, and the sensitivity of each time scale is calculated based on each anomaly score. Finally, the anomaly data detection threshold for anomaly data detection is calculated based on the maximum value of the multiple sensitivities. Compared with traditional statistical and machine learning-based anomaly detection methods, this solution does not involve a large number of training processes and statistical processes, thereby improving the detection efficiency of anomaly data and meeting the needs of real-time processing of large amounts of data.

[0080] The present application discloses a specific method for detecting abnormal data of a device. Figure 2 As shown, the method includes:

[0081] Step S21: When a preset time period is reached, historical operation data and / or historical status data of the components to be monitored in the server within a preset time period before the current moment are obtained to obtain historical time series data; the server is a server located in a preset data center.

[0082] In this embodiment, when a preset time period (e.g., daily or weekly) is reached, historical operating data (e.g., CPU utilization) and / or historical status data (e.g., CPU temperature) of a component to be monitored (e.g., a CPU) collected from a server at a preset recording period (e.g., every five minutes) within a preset time period before the current moment can be retrieved, thereby obtaining historical time series data for the CPU. The server in question is located in a preset data center. It is understood that the goal of recording CPU utilization is to detect possible abnormal load conditions, and based on this data, system failures or security issues can be predicted.

[0083] In a specific embodiment, obtaining historical operating data and / or historical status data of the monitored component in the server within a preset time period before the current moment to obtain historical time series data may include obtaining CPU usage of the server recorded according to a preset collection period within the preset time period before the current moment to obtain historical time series data. For example, obtaining CPU usage recorded every 5 minutes for a week before the current moment to obtain CPU utilization time series data.

[0084] Step S22: extracting information from the historical time series data according to the minute time scale, hour time scale, day time scale and week time scale to obtain time scale series data including minute level, hour level, day level and week level.

[0085] In this embodiment, after obtaining the historical time series data for the CPU, information extraction can be performed on the historical time series data according to the minute time scale (5 minutes), hour time scale (1 hour), day time scale (1 day), and week time scale (1 week), thereby obtaining time scale series data including minute level, hour level, day level, and week level.

[0086] Step S23: Calculate the moving average of each time scale based on the preset sliding window size and the time scale series data, and calculate the moving standard deviation of each time scale based on the moving average.

[0087] For example, obtain the original minute-level time scale series data recorded every 5 minutes, and generate hour-level time scale series data based on the minute-level time scale series data (obtained by calculating the average of 12 data points per hour), then generate day-level time scale series data based on the hour-level time scale series data (obtained by calculating the average of 24 hours of data per day), and generate weekly time scale series data based on the day-level time scale series data (obtained by calculating the average of 7 days of data per week), and then for each time scale, use a 24-hour sliding window k to calculate the moving average, and calculate the moving standard deviation of each time scale based on the moving average.

[0088] Step S24: calculating anomaly scores of each time scale based on the time scale series data, the moving standard deviation, and the moving average to obtain a plurality of first anomaly scores.

[0089] Step S25: Calculate the sensitivity of each time scale based on the first anomaly score and a preset score threshold, and calculate the abnormal data detection threshold based on the maximum value of the sensitivities.

[0090] For example, by analyzing data within a preset time period (within a week) and calculating the sensitivity of each time scale based on the sensitivity calculation formula Sensitivity_s = Σ(Score_s(t) > threshold) / T, the sensitivity of the 5-minute time scale is 0.12, the sensitivity of the hourly time scale is 0.08, the sensitivity of the daily time scale is 0.03, and the sensitivity of the weekly time scale is 0.01. Then, based on the anomaly scores of the past 7 days of 5-minute data, the mean μ=2.5 and the standard deviation σ=0.5 are calculated. The anomaly data detection threshold is calculated based on the maximum value of the above sensitivities (i.e., 0.12), and the anomaly data detection threshold Threshold_dynamic = 2.5 + 3 * 0.5 = 4 is obtained, that is, the adjustable parameter α=3.

[0091] Step S26: Based on the preset sliding window size, the anomaly score of the to-be-detected data of the target device at the current moment is calculated to obtain a current anomaly score.

[0092] Step S27: determining whether the data to be detected is abnormal data based on a comparison result between the current abnormality score and the abnormal data detection threshold.

[0093] For example, at 10:15 a.m. on a certain weekday, a data point with a CPU usage rate of 95% is detected, and its anomaly score is 4.5. By comparison, it is found that the current anomaly score of 4.5 is greater than the anomaly data detection threshold of 4. Therefore, the current data to be detected is determined to be anomaly data, and the corresponding data point is an isolated point.

[0094] Furthermore, an alarm message is generated. When the system administrator receives the alarm message, he finds through analysis that it is caused by an unplanned large-scale data processing task. He then adjusts the task scheduling to solve the problem that caused the anomaly. When the next pre-cycle arrives, such as 3:00 a.m. the following Sunday, the server system automatically triggers step S21 and re-executes the entire device abnormal data detection process. If there have been multiple similar high load situations in the past week, the system can make the following adjustments: re-evaluate the sensitivity of each time scale and obtain new results: the sensitivity of the 5-minute time scale is 0.15, the sensitivity of the hourly time scale is 0.11, the sensitivity of the daily time scale is 0.04, and the sensitivity of the weekly time scale is 0.01. At this time, the system still selects the 5-minute level as the optimal detection scale, but updates the abnormal data detection threshold with a new mean μ=3.0 and a standard deviation σ=0.6. The new abnormal data detection threshold is Threshold_dynamic = 3.0 + 3 * 0.6 = 4.8. In the following week, the system will use this new abnormal data detection threshold for outlier detection (i.e., abnormal data detection). This threshold adjustment method enables the system to adapt to recent changes in load patterns, reducing false alarms while still detecting true abnormalities. For example, on the following Tuesday, at 2:30 a.m., the system detected a data point with a CPU usage of 98% and an anomaly score of 5.2. Because 5.2 > 4.8 (the new anomaly data detection threshold), the system triggered an alarm again. After investigation, the system administrator discovered that this was caused by an abnormally time-consuming database index rebuild process. The system administrator then optimized the database to resolve the issue that caused the anomaly. This process will continue as the system constantly adapts to new data patterns and changes in the operating environment.

[0095] Furthermore, after determining whether the data to be detected is abnormal data based on the comparison result between the current anomaly score and the abnormal data detection threshold, the process may further include: if the data to be detected is abnormal data, determining whether the data to be detected is greater than a preset utilization threshold; and if the data to be detected is greater than the preset utilization threshold, triggering a load anomaly alarm. For example, if the data to be detected (CPU utilization) is abnormal data, determining whether the data to be detected (e.g., CPU utilization = 95%) is greater than a preset utilization threshold of 75%. Since 95% > 75%, a load anomaly alarm is triggered, thereby alerting the user of the current load anomaly.

[0096] For more specific processing procedures of the above steps S24 and S26, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be repeated here.

[0097] It can be seen that in an embodiment of the present application, when a preset time period is reached, historical operation data and / or historical status data of the components to be monitored in the server within a preset time period before the current moment are obtained to obtain historical time series data, and then information extraction is performed on the historical time series data according to minute time scales, hourly time scales, daily time scales and weekly time scales to obtain time scale series data including minute level, hourly level, daily level and weekly level. Then, the moving average of each time scale is calculated based on the preset sliding window size and the time scale series data, and the moving standard deviation of each time scale is calculated based on the moving average value. Then, the anomaly score of each time scale is calculated based on the time scale series data, the moving standard deviation and the moving average value to obtain multiple first anomaly scores. Finally, the sensitivity of each time scale is calculated based on the first anomaly score and the preset score threshold, and the anomaly data detection threshold for anomaly detection is calculated based on the maximum value of the sensitivities. The embodiment of the present application can capture anomalies of different types and durations in the server by performing anomaly detection simultaneously on multiple time scales. The entire process does not require human participation, the execution process is simple, and there is no need to conduct statistics or training on a large amount of data. Therefore, the calculation efficiency is high, and it can be quickly deployed and applied, while meeting the needs of real-time processing of large amounts of data. In addition, the present application will automatically trigger the abnormal data detection process whenever a preset time period is reached, and obtain a new abnormal data detection threshold. That is, the abnormal data detection threshold will be automatically updated with the preset time period, and the detection threshold that best suits the current application scenario will be calculated. Therefore, it can adapt to the characteristics of different devices and environments, ensuring long-term adaptability.

[0098] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0099] Figure 3 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the device abnormal data detection method disclosed in any of the aforementioned embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0100] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0101] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0102] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of implementing the device abnormal data detection method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program capable of implementing other specific tasks.

[0103] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned device abnormal data detection method. The specific steps of this method can be referred to the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0104] Furthermore, an embodiment of the present application also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the device abnormal data detection method disclosed above.

[0105] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0106] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0107] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0108] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0109] The above is a detailed introduction to the device abnormal data detection method, device, equipment and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for detecting abnormal data of a device, characterized in that: include: Extracting information from the historical time series data of the target device within a preset time period before the current moment according to different time scales to obtain multiple time scale series data; the historical time series data is the historical operation data and / or historical status data of the component to be monitored; Calculating anomaly scores for each time scale based on a preset sliding window size and the time scale sequence data to obtain a plurality of first anomaly scores; Calculating the sensitivity of each time scale based on the first anomaly score and a preset score threshold, and calculating the abnormal data detection threshold based on the maximum value of the sensitivities; Calculating, based on the preset sliding window size, an anomaly score of the to-be-detected data of the target device at the current moment to obtain a current anomaly score; Determining whether the data to be detected is abnormal data according to a comparison result between the current abnormality score and the abnormal data detection threshold; The method of calculating the anomaly score of each time scale based on the preset sliding window size and the time scale sequence data to obtain a plurality of first anomaly scores includes: calculating the moving average of each time scale based on the preset sliding window size and the time scale sequence data, and calculating the moving standard deviation of each time scale based on the moving average; and calculating the anomaly score of each time scale based on the time scale sequence data, the moving standard deviation, and the moving average to obtain a plurality of first anomaly scores. The calculation of the sensitivity of each time scale based on the first anomaly score and the preset score threshold includes: obtaining a score greater than the preset score threshold from multiple first anomaly scores corresponding to each time scale within the preset time length to obtain a second anomaly score, and counting the number of second anomaly scores corresponding to each time scale to obtain a second statistical result; calculating the sum of multiple second anomaly scores corresponding to each time scale to obtain a second sum; and calculating the ratio of the second sum to the second statistical result to obtain the sensitivity of the corresponding time scale.

2. The device abnormal data detection method according to claim 1, characterized in that: The calculating of the moving average of each time scale based on the preset sliding window size and the time scale series data includes: respectively calculating the sum of the plurality of data in each of the time-scale series data to obtain a first sum, and counting the number of the plurality of data in the time-scale series data to obtain a first statistical result; The ratio of the first sum value to the first statistical result is calculated to obtain a first ratio, and the moving average of each time scale is calculated based on a preset sliding window size and the first ratio using a moving average method.

3. The device abnormal data detection method according to claim 2, characterized in that: The calculating of the anomaly score of each time scale based on the time scale series data, the moving standard deviation, and the moving average to obtain a plurality of first anomaly scores includes: Calculating the difference between the first ratio and the moving average corresponding to each time scale to obtain a target difference; The absolute value of the target difference is calculated, and the ratio of the absolute value to the moving standard deviation is calculated to obtain a first anomaly score corresponding to each time scale.

4. The device abnormal data detection method according to claim 1, characterized in that: The calculating of the abnormal data detection threshold based on the maximum value of the sensitivities includes: Determine a maximum value among the multiple sensitivities corresponding to multiple time scales, and calculate the mean and standard deviation of the first anomaly score corresponding to the maximum value within the preset time length to obtain a target mean and a target standard deviation; determining a target time scale corresponding to the maximum value, and obtaining a preset adjustable parameter corresponding to the target time scale; The product of the target standard deviation and a preset adjustable parameter is calculated, and the sum of the product and the target mean is calculated to obtain an abnormal data detection threshold.

5. The device abnormal data detection method according to claim 4, characterized in that: The determining whether the data to be detected is abnormal data according to a comparison result between the current abnormality score and the abnormal data detection threshold includes: If the current anomaly score is greater than the abnormal data detection threshold, the data to be detected is determined to be abnormal data; If the current anomaly score is not greater than the abnormal data detection threshold, the data to be detected is determined to be normal data.

6. The device abnormal data detection method according to claim 5, characterized in that: After determining whether the data to be detected is abnormal data according to the comparison result between the current abnormality score and the abnormal data detection threshold, the method further includes: If the data to be detected is abnormal data, the number of abnormal data determined to be abnormal data within a preset time is counted to obtain a first number; Obtaining the number of false alarms in the abnormal data determined within the preset time to obtain a second number; Calculating a ratio of the second number to the first number to obtain a second ratio, and determining whether the second ratio is greater than a preset false alarm threshold; If the second ratio is greater than the preset false alarm threshold, the value of the preset adjustable parameter is adjusted down, and the step of extracting information from the historical time series data of the target device within the preset time length before the current moment according to different time scales is triggered to obtain multiple time scale series data.

7. The device abnormal data detection method according to claim 1, characterized in that: The information is extracted from the historical time series data of the target device within a preset time period before the current moment according to different time scales to obtain multiple time scale series data, including: When a preset time period is reached, historical operation data and / or historical status data of the components to be monitored in the server within a preset time period before the current moment are obtained to obtain historical time series data; the server is a server located in a preset data center; Information is extracted from the historical time series data according to minute time scales, hour time scales, day time scales, and week time scales to obtain time scale series data including minute level, hour level, day level, and week level.

8. The device abnormal data detection method according to claim 7, characterized in that: The acquiring of historical operation data and / or historical status data of the component to be monitored in the server within a preset time period before the current moment to obtain historical time series data includes: Get the CPU usage of the server recorded according to the preset collection period within the preset time period before the current time to obtain historical time series data.

9. The device abnormal data detection method according to claim 7, characterized in that: After determining whether the data to be detected is abnormal data according to the comparison result between the current abnormality score and the abnormal data detection threshold, the method further includes: If the data to be detected is abnormal data, determining whether the data to be detected is greater than a preset usage rate threshold; If the data to be detected is greater than the preset usage rate threshold, a load abnormality alarm is triggered.

10. The device abnormal data detection method according to any one of claims 1 to 9, characterized in that: After determining whether the data to be detected is abnormal data according to the comparison result between the current abnormality score and the abnormal data detection threshold, the method further includes: If the data to be detected is abnormal data, alarm information including the abnormal data, the abnormal data detection threshold and the current abnormality score is generated, so as to process the situation causing the abnormal data.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the device abnormal data detection method according to any one of claims 1 to 10 is implemented.

12. An electronic device, characterized in that: The device comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the device abnormal data detection method according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, the device abnormal data detection method according to any one of claims 1 to 10 is implemented.

Images