Equipment abnormal data detection method and device, equipment and storage medium

Abstract

The invention discloses an equipment abnormal data detection method and device, equipment and a storage medium, and relates to the technical field of computers, and the method comprises the steps: carrying out the information extraction of historical time series data of target equipment within a preset duration before a current moment according to different time scales, and obtaining a plurality of time scale series data; calculating an abnormal score of each time scale based on a preset sliding window size and the time scale sequence data to obtain a plurality of first abnormal scores; calculating the sensitivity of each time scale based on the first abnormal score and a preset score threshold, and calculating an abnormal data detection threshold based on the maximum value in the sensitivity; calculating an abnormal score of the to-be-detected data of the target equipment at the current moment based on the size of a preset sliding window to obtain a current abnormal score; and determining whether the to-be-detected data is abnormal data or not according to a comparison result between the current abnormal score and an abnormal data detection threshold. The abnormal data detection efficiency can be improved, and the requirement for processing a large amount of data in real time is met.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for detecting abnormal data of a device. Background Art

[0002] With the rapid development of data collection technology, the amount of time series data that can be obtained in the fields of industrial Internet of Things, intelligent manufacturing and equipment health management is also increasing. These data reflect the operating status of the equipment. However, how to accurately identify anomalies or isolated points from these massive data and timely warn of potential failures is still a problem that needs to be solved.

[0003] For the detection of abnormal data in massive data, traditional detection methods are usually based on a single time scale for analysis, which may lead to the neglect of certain types of anomalies. For example, some anomalies may not be obvious on an hourly scale, but very significant on a daily or weekly scale. In addition, different devices may have different characteristics and are suitable for anomaly detection on different time scales. Therefore, a method that can adaptively select the optimal detection scale becomes particularly important.

[0004] The current mainstream device abnormal data detection methods, such as statistical anomaly detection methods and machine learning-based anomaly detection methods, also have some defects. For example, the moving average method and standard deviation method in statistical anomaly detection methods identify anomalies by calculating the deviation between a data point and the average value of its surrounding data, but usually only analyze on a single time scale (i.e., the time scale is fixed), so anomalies on other scales may be ignored. In addition, the fixed threshold used in this method may not adapt to the dynamic changes of data, which may lead to false positives or false negatives. In addition, it is impossible to automatically adjust the detection strategy according to the characteristics of different devices, so it lacks adaptive capabilities. For example, the isolation forest method in machine learning-based anomaly detection methods Forest algorithm, this method identifies anomalies by constructing a decision tree. Specifically, it constructs a decision tree by randomly selecting features and split points, and repeats this step many times to build a decision tree forest. Then, the anomaly score of each data point is calculated, and based on its average path length in the decision tree and a preset threshold, the points with scores higher than the threshold are marked as anomalies. However, this method has high computational complexity and may not meet the requirements of real-time processing for large-scale real-time data. It also requires a large amount of training data and may not perform well when data is scarce or a new system is deployed. In addition, it lacks interpretability and is difficult to explain why a certain point is judged to be abnormal, which is not conducive to subsequent fault diagnosis. It cannot effectively utilize the characteristics of time series and ignores the temporal information and multi-scale characteristics of the data. Summary of the invention

[0005] In view of this, the purpose of this application is to provide a device abnormal data detection method, device, equipment and storage medium, which can capture abnormalities of different types and durations, improve the detection efficiency of abnormal data, and meet the needs of real-time processing of large amounts of data. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses a method for detecting abnormal data of a device, comprising:

[0007] According to different time scales, information is extracted from the historical time series data of the target device within a preset time period before the current moment to obtain multiple time scale series data;

[0008] Based on the preset sliding window size and the time scale sequence data, respectively calculate the anomaly score of each time scale to obtain a plurality of first anomaly scores;

[0009] Calculating the sensitivity of each time scale based on the first anomaly score and a preset score threshold, and calculating the abnormal data detection threshold based on the maximum value of the sensitivities;

[0010] Based on the preset sliding window size, calculating the anomaly score of the to-be-detected data of the target device at the current moment to obtain a current anomaly score;

[0011] Whether the data to be detected is abnormal data is determined according to a comparison result between the current abnormality score and the abnormal data detection threshold.

[0012] Optionally, the calculating of the anomaly scores of each time scale based on the preset sliding window size and the time scale sequence data to obtain multiple first anomaly scores includes:

[0013] Calculate the moving average of each time scale based on the preset sliding window size and the time scale series data, and calculate the moving standard deviation of each time scale based on the moving average;

[0014] The abnormality score of each time scale is calculated based on the time scale series data, the moving standard deviation and the moving average to obtain a plurality of first abnormality scores.

[0015] Optionally, the calculating the moving average of each time scale based on the preset sliding window size and the time scale series data respectively includes:

[0016] respectively calculating the sum of the plurality of data in each of the time scale series data to obtain a first sum, and counting the number of the plurality of data in the time scale series data to obtain a first statistical result;

[0017] The ratio of the first sum value to the first statistical result is calculated to obtain a first ratio, and the moving average value of each time scale is calculated based on a preset sliding window size and the first ratio by using a moving average method.

[0018] Optionally, the calculating the anomaly score of each time scale based on the time scale series data, the moving standard deviation and the moving average to obtain multiple first anomaly scores includes:

[0019] Calculate the difference between the first ratio and the moving average value corresponding to each time scale to obtain a target difference;

[0020] The absolute value of the target difference is calculated, and the ratio of the absolute value to the moving standard deviation is calculated to obtain a first anomaly score corresponding to each time scale.

[0021] Optionally, the calculating the sensitivity of each time scale based on the first anomaly score and a preset score threshold includes:

[0022] Obtaining scores greater than a preset score threshold from among the plurality of first anomaly scores corresponding to each time scale within the preset duration to obtain a second anomaly score, and counting the number of the second anomaly scores corresponding to each time scale to obtain a second statistical result;

[0023] Calculating a sum of a plurality of the second anomaly scores corresponding to each time scale to obtain a second sum;

[0024] The ratio of the second sum value to the second statistical result is calculated to obtain the sensitivity of the corresponding time scale.

[0025] Optionally, the calculating the abnormal data detection threshold based on the maximum value of the sensitivities includes:

[0026] Determine a maximum value among the multiple sensitivities corresponding to the multiple time scales, and calculate a mean and a standard deviation of the first anomaly score corresponding to the maximum value within the preset time length to obtain a target mean and a target standard deviation;

[0027] Determining a target time scale corresponding to the maximum value, and obtaining a preset adjustable parameter corresponding to the target time scale;

[0028] The product of the target standard deviation and a preset adjustable parameter is calculated, and the sum of the product and the target mean is calculated to obtain an abnormal data detection threshold.

[0029] Optionally, determining whether the data to be detected is abnormal data according to a comparison result between the current abnormal score and the abnormal data detection threshold includes:

[0030] If the current anomaly score is greater than the abnormal data detection threshold, the data to be detected is determined to be abnormal data;

[0031] If the current abnormality score is not greater than the abnormal data detection threshold, the data to be detected is determined to be normal data.

[0032] Optionally, after determining whether the data to be detected is abnormal data according to the comparison result between the current abnormal score and the abnormal data detection threshold, the method further includes:

[0033] If the data to be detected is abnormal data, the number of the data determined to be abnormal data within a preset time is counted to obtain a first number;

[0034] Obtaining the number of false alarms in the abnormal data determined within the preset time to obtain a second number;

[0035] Calculating a ratio of the second number to the first number to obtain a second ratio, and determining whether the second ratio is greater than a preset false alarm threshold;

[0036] If the second ratio is greater than the preset false alarm threshold, the value of the preset adjustable parameter is reduced, and the step of extracting information from the historical time series data of the target device within a preset time length before the current moment according to different time scales to obtain multiple time scale series data is triggered.

[0037] Optionally, the information is extracted from the historical time series data of the target device within a preset time period before the current moment according to different time scales to obtain multiple time scale series data, including:

[0038] When a preset time period is reached, historical operation data and / or historical status data of the components to be monitored in the server within a preset time period before the current moment are obtained to obtain historical time series data; the server is a server located in a preset data center;

[0039] Information is extracted from the historical time series data according to the minute time scale, hour time scale, day time scale and week time scale to obtain time scale series data including minute level, hour level, day level and week level.

[0040] Optionally, the acquiring of historical operation data and / or historical status data of the component to be monitored in the server within a preset time period before the current moment to obtain historical time series data includes:

[0041] Get the CPU usage of the server recorded according to the preset collection period within the preset time before the current time, and obtain the historical time series data.

[0042] Optionally, after determining whether the data to be detected is abnormal data according to the comparison result between the current abnormal score and the abnormal data detection threshold, the method further includes:

[0043] If the data to be detected is abnormal data, determining whether the data to be detected is greater than a preset usage rate threshold;

[0044] If the data to be detected is greater than the preset usage rate threshold, a load abnormality alarm is triggered.

[0045] Optionally, after determining whether the data to be detected is abnormal data according to the comparison result between the current abnormal score and the abnormal data detection threshold, the method further includes:

[0046] If the data to be detected is abnormal data, alarm information including the abnormal data, the abnormal data detection threshold and the current abnormal score is generated so as to process the situation causing the abnormal data.

[0047] In a second aspect, the present application discloses a computer program product, including a computer program / instruction, which implements the aforementioned device abnormal data detection method when executed by a processor.

[0048] In a third aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein the processor implements the aforementioned device abnormal data detection method when executing a computer program stored in the memory.

[0049] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned device abnormal data detection method is implemented.

[0050] It can be seen that the present application first extracts information from the historical time series data of the target device within a preset time length before the current moment according to different time scales to obtain multiple time scale series data, and then calculates the anomaly score of each time scale based on the preset sliding window size and the time scale series data to obtain multiple first anomaly scores, and then calculates the sensitivity of each time scale based on the first anomaly score and the preset score threshold, and calculates the anomaly data detection threshold based on the maximum value of the sensitivities, and then calculates the anomaly score of the data to be detected of the target device at the current moment based on the preset sliding window size to obtain the current anomaly score, and determines whether the data to be detected is abnormal data based on the comparison result between the current anomaly score and the anomaly data detection threshold. The present application extracts information from the historical time series data of the target device within a preset time period before the current moment according to different time scales, thereby obtaining sequence data of multiple different time scales. By converting a single time series data into sequence data of multiple different time scales, it is possible to capture anomalies of different types and durations, such as short-term burst loads and long-term performance degradation trends; further, the anomaly score of each time scale is calculated based on the sequence data corresponding to each time scale and the preset sliding window size, and the sensitivity of each time scale is calculated based on each anomaly score, and finally the anomaly data detection threshold for anomaly data detection is calculated based on the maximum value of the multiple sensitivities. Compared with traditional statistical and machine learning-based anomaly detection methods, this solution does not involve a large number of training processes and statistical processes, thereby improving the detection efficiency of anomaly data and meeting the needs of real-time processing of large amounts of data.

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0052] Figure 1 A flow chart of a method for detecting abnormal data of a device disclosed in this application;

[0053] Figure 2 A flowchart of a specific device abnormal data detection method disclosed in this application;

[0054] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] The present application discloses a method for detecting abnormal data of a device. Figure 1 As shown, the method includes:

[0057] Step S11: extracting information from the historical time series data of the target device within a preset time period before the current moment according to different time scales to obtain multiple time scale series data.

[0058] In this embodiment, when it is necessary to perform abnormal data detection on the target device, the historical time series data of the target device within a preset time period before the current moment (such as within the past week) is first obtained, and then information is extracted from the above historical time series data according to different time scales (such as minute time scale, hour time scale, day time scale, week time scale, month time scale, etc.), so as to obtain multiple time scale sequence data; wherein, the target device includes but is not limited to servers, computers and other devices, and the historical time series data can specifically be the historical time series data of the components to be monitored in the target device, such as the CPU (central processing unit, Central Processing Unit) usage rate, CPU temperature, memory usage rate, fan temperature or fan speed data recorded every 5 minutes in the server. By extracting sequence data of multiple time scales from the historical time series data, abnormal patterns of different types and durations can be captured.

[0059] For example, the fan temperature series data recorded by the server system according to the preset monitoring cycle (every 5 minutes) in the past week are obtained, and then the information of the above fan temperature series data is extracted according to the hourly time scale, the daily time scale, and the weekly time scale to obtain multiple time scale series data; among which, the time scale series data corresponding to the hourly time scale is the fan temperature data extracted with hours as the information extraction unit. Since the preset monitoring cycle is to collect data every 5 minutes, there are 12 data points extracted in one hour, and the average of these 12 data can be calculated as one valid data point corresponding to the hourly time scale; similarly, the data points extracted in one day are 12*24, and the average of these 12*24 data can be calculated to obtain one valid data point corresponding to the daily time scale; similarly, the data points extracted in one week are 12*24*7, and the average of these 12*24*7 data can be calculated to obtain one valid data point corresponding to the weekly time scale.

[0060] Step S12: Calculate the anomaly score of each time scale based on the preset sliding window size and the time scale series data to obtain a plurality of first anomaly scores.

[0061] In this embodiment, after extracting information from the historical time series data of the target device within a preset time length before the current moment according to different time scales to obtain multiple time scale series data, further, the anomaly score of each time scale can be calculated based on the preset sliding window size k and the above-mentioned time scale series data to obtain multiple corresponding first anomaly scores.

[0062] In a specific implementation, the abnormality score of each time scale is calculated based on the preset sliding window size and the time scale sequence data to obtain multiple first abnormality scores, which may include: calculating the moving average of each time scale based on the preset sliding window size and the time scale sequence data, and calculating the moving standard deviation of each time scale based on the moving average; calculating the abnormality score of each time scale based on the time scale sequence data, the moving standard deviation and the moving average to obtain multiple first abnormality scores. In this embodiment, the moving average of each time scale can be calculated based on a preset sliding window size k and the time scale sequence data, and then the moving standard deviation of each time scale is calculated based on the above moving average, and then the abnormality score of each time scale is calculated based on the above time scale sequence data corresponding to each time scale, the above moving standard deviation and the above moving average, so as to obtain the abnormality score of all time scales.

[0063] Specifically, the calculation of the moving average of each time scale based on the preset sliding window size and the time scale series data may include: calculating the sum of multiple data in each time scale series data to obtain a first sum, and counting the number of multiple data in the time scale series data to obtain a first statistical result; calculating the ratio of the first sum to the first statistical result to obtain a first ratio, and using the moving average method to calculate the moving average of each time scale based on the preset sliding window size and the first ratio. For example, when the time scale series data is recorded every 5 minutes, information can be extracted from the above time scale series data according to the minute time scale, hour time scale, day time scale, and week time scale to obtain multiple time scale series data; wherein the minute time scale is the time series data originally recorded by the system, recorded as X_m(t) for t in minute; then, based on X_m(t), the sum of multiple data in the time scale series data corresponding to the hourly time scale, the daily time scale, and the weekly time scale (i.e., the above-mentioned valid data points) is calculated respectively, and the number of multiple data in the time scale series data is counted to obtain the statistical result, and then the ratio of the sum to the statistical result is calculated; wherein, the ratio calculation formula corresponding to the hourly time scale is X_h(t) = Σ(X_m(t)for t in hour) / 12, the ratio calculation formula corresponding to the day time scale is X_d(t) = Σ(X_h(t) for t inday) / 24, and the ratio calculation formula corresponding to the weekly time scale is X_w(t) = Σ(X_d(t) for t in week) / 7. Furthermore, the moving average method is used to calculate the moving average of each time scale based on the preset sliding window size k (such as k is 24 hours) and the above ratio. Specifically, the time scale can be represented as s in {m, h, d, w}, and then the moving average of each time scale MA_s(t) = Σ(X_s(i) for iin [tk, t+k]) / (2k+1) is calculated, and the moving standard deviation of each time scale SD_s(t) = sqrt(Σ((X_s(i) - MA_s(t))^2 for i in [tk, t+k]) / (2k+1)) is calculated based on the moving average MA_s(t).

[0064] Specifically, the calculation of the anomaly score of each time scale based on the time scale series data, the moving standard deviation and the moving average value to obtain multiple first anomaly scores may include: calculating the difference between the first ratio corresponding to each time scale and the moving average value to obtain a target difference; calculating the absolute value of the target difference, and calculating the ratio of the absolute value to the moving standard deviation to obtain the first anomaly score corresponding to each time scale. That is, the difference between the first ratio corresponding to each time scale and the moving average value is calculated, and the absolute value of the difference is calculated, and then the ratio of the absolute value to the above-mentioned moving standard deviation is calculated, so as to obtain the first anomaly score corresponding to each time scale. Specifically, the calculation formula of the first anomaly score Score_s(t) is: Score_s(t) = |X_s(t) - MA_s(t)| / SD_s(t).

[0065] Step S13: calculating the sensitivity of each time scale based on the first abnormality score and a preset score threshold, and calculating the abnormal data detection threshold based on the maximum value of the sensitivities.

[0066] In this embodiment, after calculating the anomaly scores of each time scale based on the preset sliding window size and the time scale series data, the sensitivity of each time scale can be calculated based on the above-mentioned first anomaly score and the preset score threshold, and then the maximum value of all sensitivities is determined, and the threshold for abnormal data detection is calculated based on the maximum value.

[0067] It should be pointed out that in order to improve the accuracy of abnormal data detection, before performing abnormal data detection on the device, different adjustable parameters can be set for different time scales using the grid search method in advance, so as to obtain preset adjustable parameters corresponding to each time scale.

[0068] Specifically, the calculation of the sensitivity of each time scale based on the first abnormality score and the preset score threshold, and the calculation of the abnormal data detection threshold based on the maximum value of the sensitivity, may include: obtaining the scores greater than the preset score threshold from the multiple first abnormality scores corresponding to each time scale within the preset time length to obtain a second abnormality score, and counting the number of the second abnormality scores corresponding to each time scale to obtain a second statistical result; calculating the sum of the multiple second abnormality scores corresponding to each time scale to obtain a second sum; calculating the ratio of the second sum to the second statistical result to obtain the sensitivity of the corresponding time scale. In this embodiment, the scores greater than a preset score threshold (threshold) among multiple first abnormality scores Score_s(t) corresponding to each time scale within a preset time length (such as within a week) are first obtained to obtain a second abnormality score, and then the number of all second abnormality scores corresponding to each time scale is counted to obtain a second statistical result T; then, the sum of the multiple second abnormality scores corresponding to each time scale is calculated to obtain a second sum, and the ratio of the second sum to the second statistical result T is calculated to obtain the sensitivity Sensitivity_s of the corresponding time scale. The specific calculation formula is: Sensitivity_s = Σ(Score_s(t) > threshold) / T.

[0069] Furthermore, the calculation of the abnormal data detection threshold based on the maximum value among the sensitivities may specifically include: determining the maximum value among multiple sensitivities corresponding to multiple time scales, and calculating the mean and standard deviation of the first abnormality score corresponding to the maximum value within the preset time length to obtain a target mean and a target standard deviation; determining the target time scale corresponding to the maximum value, and obtaining a preset adjustable parameter corresponding to the target time scale; calculating the product of the target standard deviation and the preset adjustable parameter, and calculating the sum of the product and the target mean to obtain the abnormal data detection threshold. In this embodiment, the maximum value Best_scale of all sensitivities corresponding to multiple time scales can be determined first. The specific calculation formula is: Best_scale = argmax(Sensitivity_s for s in {m, h, d, w}), and then the mean and standard deviation of the first abnormal score Score_s(t) corresponding to the maximum value Best_scale within a preset time length (such as within one week) are calculated to obtain the target mean μ and the target standard deviation σ. Finally, the target time scale corresponding to the maximum value Best_scale is determined, and the preset adjustable parameter α corresponding to the target time scale is obtained. Then, the product of the target standard deviation σ and the preset adjustable parameter α is calculated, and then the sum of the product and the target mean μ is calculated to obtain the abnormal data detection threshold Threshold_dynamic(t). The specific calculation formula is: Threshold_dynamic(t) = μ + α * σ.

[0070] It should be pointed out that the preset score threshold (threshold) is obtained by optimizing the initial score threshold threshold_initial, and the setting of the initial score threshold threshold_initial can be determined by a variety of methods. For example, according to the experience of the system administrator or domain expert (i.e., domain knowledge), a reasonable initial score threshold is set for each time scale; for example, time series data before the historical moment is collected, and an initial threshold is set by analyzing the data; for example, the initial score threshold is directly set to a higher value to avoid too many false alarms, and then the value is gradually adjusted as the system runs; or time series data before a historical moment is collected, and then the mean μ and standard deviation σ are calculated using the data, and then the initial score threshold is set. The specific calculation formula is: threshold_initial = μ + g * σ. In a specific implementation, g=2 or g=3, and the value depends on the required sensitivity_s.

[0071] In addition, it should be pointed out that the preset adjustable parameter α is an adjustable parameter used to control the impact of the anomaly score on the final result in the sensitivity calculation. The parameter value can be determined by a variety of methods. For example, based on the experience of similar systems, an initial value such as 0.1 or 0.2 is set; for example, it can be set according to the system's sensitivity to anomalies. For systems that need to be more sensitive, α can be set larger; for example, it can be set through a preset dynamic adjustment mechanism, which allows α to be dynamically adjusted according to system performance. If the false alarm rate is found to be too high, the value of α is gradually reduced; for example, in the system initialization stage, a small part of the data, such as historical time series data within the past day, can be used to find a better α value through grid search (GridSearch). In addition, in order to adapt to the characteristics of different time scales, different α values ​​can also be set for different time scales.

[0072] Step S14: Based on the preset sliding window size, the anomaly score of the to-be-detected data of the target device at the current moment is calculated to obtain a current anomaly score.

[0073] In this embodiment, after obtaining the abnormal data detection threshold, the data to be detected of the target device at the current moment, such as the temperature data of the fan, the CPU usage rate, etc., is obtained, and then the abnormal score of the current data to be detected is calculated based on the preset sliding window size k to obtain the current abnormal score. It should be noted that the calculation process of the current abnormal score is the same as the calculation method of the abnormal score in step S12, which will not be repeated here.

[0074] Step S15: determining whether the data to be detected is abnormal data according to a comparison result between the current abnormal score and the abnormal data detection threshold.

[0075] In this embodiment, after the abnormality score of the current data to be detected is calculated, it can be directly determined whether the current data to be detected is abnormal data according to the comparison result between the current abnormality score and the abnormal data detection threshold.

[0076] Specifically, determining whether the data to be detected is abnormal data according to the comparison result between the current abnormal score and the abnormal data detection threshold may include: if the current abnormal score is greater than the abnormal data detection threshold, it is determined that the data to be detected is abnormal data; if the current abnormal score is not greater than the abnormal data detection threshold, it is determined that the data to be detected is normal data. That is, it is judged whether the abnormal score of the data to be detected is greater than the abnormal data detection threshold. If it is greater (i.e., Score_Best_scale(t) > Threshold_dynamic(t)), it is directly determined that the current data to be detected is abnormal data; if it is not less than (i.e., Score_Best_scale(t) < Threshold_dynamic(t) or Score_Best_scale(t) = Threshold_dynamic(t)), it is determined that the current data to be detected is normal data.

[0077] Further, after determining whether the data to be detected is abnormal data according to the comparison result between the current abnormal score and the abnormal data detection threshold, it may further include: if the data to be detected is abnormal data, count the number of data determined to be the abnormal data within a preset time to obtain a first number; obtain the number of false alarms among the data determined to be the abnormal data within the preset time to obtain a second number; calculate the ratio of the second number to the first number to obtain a second ratio, and judge whether the second ratio is greater than a preset false alarm threshold; if the second ratio is greater than the preset false alarm threshold, reduce the value of the preset adjustable parameter, and trigger the step of extracting information from the historical time series data of the target device within a preset duration before the current moment according to different time scales to obtain multiple time scale series data. That is, after determining that the current data to be detected is abnormal data, count the number of data determined to be the abnormal data within a preset time to obtain a first number, then obtain the number of false alarms among the data determined to be abnormal data within the preset time to obtain a second number, and then calculate the ratio of the second number to the first number to obtain a second ratio. If the second ratio is greater than the preset false alarm threshold, reduce the value of the preset adjustable parameter α, and directly perform a new abnormal data check to calculate a new abnormal data detection threshold and detect the current data to be detected based on the new threshold. By continuously adjusting the preset adjustable parameter α that causes the false alarm rate to increase during the real-time detection process, the characteristics of different time scales can be adapted.

[0078] Furthermore, after determining whether the data to be detected is abnormal data according to the comparison result between the current abnormal score and the abnormal data detection threshold, it may also include: if the data to be detected is abnormal data, generating an alarm message including the abnormal data, the abnormal data detection threshold and the current abnormal score, so as to handle the situation causing the abnormal data. In this embodiment, in order to achieve the interpretability of abnormal data detection, each step of the operation may be marked with a clear physical meaning to facilitate the user to understand and interpret the detection results. After determining that the current data to be detected is abnormal data, an alarm message including the abnormal data, the abnormal data detection threshold and the current abnormal score may be generated, so that the user can know the specific cause of the abnormality, so as to handle the situation causing the abnormal data in a timely manner.

[0079] It can be seen that the embodiment of the present application first extracts information from the historical time series data of the target device within a preset time length before the current moment according to different time scales to obtain multiple time scale series data, and then calculates the anomaly score of each time scale based on the preset sliding window size and the time scale series data to obtain multiple first anomaly scores, and then calculates the sensitivity of each time scale based on the first anomaly score and the preset score threshold, and calculates the anomaly data detection threshold based on the maximum value of the sensitivities, and then calculates the anomaly score of the data to be detected of the target device at the current moment based on the preset sliding window size to obtain the current anomaly score, and determines whether the data to be detected is abnormal data based on the comparison result between the current anomaly score and the anomaly data detection threshold. The embodiment of the present application extracts information from the historical time series data of the target device within a preset time period before the current moment according to different time scales, thereby obtaining sequence data of multiple different time scales. By converting a single time series data into sequence data of multiple different time scales, it is possible to capture anomalies of different types and durations, such as short-term burst loads and long-term performance degradation trends. Furthermore, the anomaly score of each time scale is calculated based on the sequence data corresponding to each time scale and the preset sliding window size, and the sensitivity of each time scale is calculated based on each anomaly score. Finally, the anomaly data detection threshold for anomaly data detection is calculated based on the maximum value of the multiple sensitivities. Compared with traditional statistical and machine learning-based anomaly detection methods, this solution does not involve a large number of training processes and statistical processes, thereby improving the detection efficiency of anomaly data and meeting the needs of real-time processing of large amounts of data.

[0080] The present application discloses a specific method for detecting abnormal data of a device. Figure 2 As shown, the method includes:

[0081] Step S21: When a preset time period is reached, historical operation data and / or historical status data of the components to be monitored in the server within a preset time period before the current moment are obtained to obtain historical time series data; the server is a server located in a preset data center.

[0082] In this embodiment, when a preset time period (such as daily or weekly) is reached, the historical operation data (such as CPU utilization) and / or historical status data (such as CPU temperature) of the component to be monitored (such as CPU) collected in the server according to the preset recording period (such as every 5 minutes) within the preset time period before the current moment can be obtained, so as to obtain the historical time series data for the CPU; wherein the server is a server located in a preset data center. It can be understood that the goal of recording CPU utilization is to detect possible abnormal load conditions, and system failures or safety issues can be predicted based on this data.

[0083] In a specific implementation, the acquisition of historical operation data and / or historical status data of the components to be monitored in the server within a preset time period before the current moment to obtain historical time series data may include: acquiring the CPU usage in the server recorded according to a preset collection period within a preset time period before the current moment to obtain historical time series data. For example, the CPU utilization recorded every 5 minutes within a week before the current moment is acquired to obtain the CPU utilization time series data.

[0084] Step S22: extracting information from the historical time series data according to the minute time scale, hour time scale, day time scale and week time scale to obtain time scale series data including minute level, hour level, day level and week level.

[0085] In this embodiment, after obtaining the historical time series data for the CPU, information extraction can be performed on the historical time series data according to the minute time scale (5 minutes), hour time scale (1 hour), day time scale (1 day) and week time scale (1 week), thereby obtaining time scale series data including minute level, hour level, day level and week level.

[0086] Step S23: Calculate the moving average of each time scale based on the preset sliding window size and the time scale series data, and calculate the moving standard deviation of each time scale based on the moving average.

[0087] For example, obtain the original minute-level time scale series data recorded every 5 minutes, and generate hour-level time scale series data based on the minute-level time scale series data (obtained by calculating the average of 12 data points per hour), then generate day-level time scale series data based on the hour-level time scale series data (obtained by calculating the average of 24 hours of data per day), and generate weekly time scale series data based on the day-level time scale series data (obtained by calculating the average of 7 days of data per week), and then for each time scale, use a 24-hour sliding window k to calculate the moving average, and calculate the moving standard deviation of each time scale based on the moving average.

[0088] Step S24: calculating the anomaly score of each time scale based on the time scale series data, the moving standard deviation and the moving average, to obtain a plurality of first anomaly scores.

[0089] Step S25: Calculate the sensitivity of each time scale based on the first abnormality score and the preset score threshold, and calculate the abnormal data detection threshold based on the maximum value of the sensitivities.

[0090] For example, by analyzing the data within a preset time length (within a week), and calculating the sensitivity of each time scale based on the sensitivity calculation formula Sensitivity_s = Σ(Score_s(t) > threshold) / T, the sensitivity of the 5-minute time scale is 0.12, the sensitivity of the hourly time scale is 0.08, the sensitivity of the daily time scale is 0.03, and the sensitivity of the weekly time scale is 0.01. Then, based on the anomaly scores of the past 7 days of 5-minute data, the mean μ=2.5 and the standard deviation σ=0.5 are calculated, and then the abnormal data detection threshold is calculated based on the maximum value of the above sensitivities (i.e. 0.12), and the abnormal data detection threshold Threshold_dynamic = 2.5 + 3 * 0.5 = 4 is obtained, that is, the adjustable parameter α=3.

[0091] Step S26: Based on the preset sliding window size, the anomaly score of the to-be-detected data of the target device at the current moment is calculated to obtain a current anomaly score.

[0092] Step S27: determining whether the data to be detected is abnormal data according to a comparison result between the current abnormal score and the abnormal data detection threshold.

[0093] For example, at 10:15 am on a certain weekday, a data point with a CPU usage rate of 95% is detected, and its anomaly score is 4.5. By comparison, it is found that the current anomaly score of 4.5 is greater than the anomaly data detection threshold of 4, so the current data to be detected is determined to be abnormal data, and the corresponding data point is an isolated point.

[0094] Furthermore, an alarm message is generated. When the system administrator receives the alarm message, he finds through analysis that it is caused by an unplanned large-scale data processing task. He then adjusts the task scheduling to solve the problem that caused the abnormality. When the next pre-cycle is reached, such as 3 a.m. next Sunday, the server system automatically triggers step S21 and re-executes the entire device abnormal data detection process. If there have been multiple similar high load conditions in the past week, the system can make the following adjustments: re-evaluate the sensitivity of each time scale and obtain new results: the sensitivity of the 5-minute time scale is 0.15, the sensitivity of the hourly time scale is 0.11, the sensitivity of the daily time scale is 0.04, and the sensitivity of the weekly time scale is 0.01. At this time, the system still selects the 5-minute level as the best detection scale, but updates the abnormal data detection threshold. The new mean μ=3.0 and the standard deviation σ=0.6, then the new abnormal data detection threshold is Threshold_dynamic = 3.0 + 3 * 0.6 =4.8. In the next week, the system will use this new abnormal data detection threshold for outlier detection (i.e., abnormal data detection). This threshold adjustment method enables the system to adapt to recent changes in load patterns, reduce false alarms, and still detect real abnormalities. For example, on the next Tuesday, the system detected a data point with a CPU usage of 98% at 2:30 a.m., with an anomaly score of 5.2. Since 5.2 > 4.8 (the new anomaly data detection threshold), the system triggered the alarm again. After investigation, the system administrator found that this was an anomaly caused by an abnormally time-consuming database index rebuild process. The system administrator then optimized the database to solve the problem that caused the anomaly. This process will continue, and the system will continue to adapt to new data patterns and changes in the operating environment.

[0095] In addition, after determining whether the data to be detected is abnormal data according to the comparison result between the current abnormal score and the abnormal data detection threshold, it can also include: if the data to be detected is abnormal data, then judging whether the data to be detected is greater than a preset usage threshold; if the data to be detected is greater than the preset usage threshold, then triggering a load abnormality alarm. For example, when the data to be detected (CPU usage) is abnormal data, judging whether the data to be detected (such as CPU usage = 95%) is greater than the preset usage threshold of 75%. Since 95%>75%, a load abnormality alarm is triggered, thereby reminding that there is currently a load abnormality phenomenon.

[0096] Among them, for more specific processing procedures of the above steps S24 and S26, reference can be made to the corresponding contents disclosed in the above embodiments, which will not be repeated here.

[0097] It can be seen that in an embodiment of the present application, when a preset time period is reached, historical operation data and / or historical status data of the components to be monitored in the server within a preset time length before the current moment are obtained to obtain historical time series data, and then information is extracted from the historical time series data according to minute time scales, hourly time scales, daily time scales and weekly time scales to obtain time scale series data including minute level, hourly level, daily level and weekly level. Then, based on the preset sliding window size and the time scale series data, the moving average of each time scale is calculated respectively, and the moving standard deviation of each time scale is calculated based on the moving average value, and then the anomaly score of each time scale is calculated based on the time scale series data, the moving standard deviation and the moving average value to obtain multiple first anomaly scores, and finally the sensitivity of each time scale is calculated based on the first anomaly score and the preset score threshold, and the anomaly data detection threshold for anomaly detection is calculated based on the maximum value of the sensitivity. The embodiments of the present application can capture anomalies of different types and durations in the server by performing anomaly detection on multiple time scales simultaneously, and the entire process does not require human intervention, the execution process is simple, and there is no need to conduct statistics or training on a large amount of data. Therefore, the calculation efficiency is high, and it can be quickly deployed and applied, while meeting the needs of real-time processing of large amounts of data. In addition, the present application will automatically trigger the abnormal data detection process whenever a preset time period is reached, and obtain a new abnormal data detection threshold, that is, the abnormal data detection threshold will be automatically updated with the preset time period, and the detection threshold that best suits the current application scenario is calculated, so it can adapt to the characteristics of different devices and environments, ensuring long-term adaptability.

[0098] Furthermore, the present application also discloses an electronic device. Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0099] Figure 3 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the device abnormal data detection method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0100] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0101] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0102] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the device abnormal data detection method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0103] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the device abnormal data detection method disclosed above is implemented. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.

[0104] Furthermore, an embodiment of the present application also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the device abnormal data detection method disclosed above.

[0105] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0106] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0107] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0108] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0109] The above is a detailed introduction to a device abnormal data detection method, device, equipment and storage medium provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for detecting abnormal data of a device, characterized in that: include: According to different time scales, information is extracted from the historical time series data of the target device within a preset time period before the current moment to obtain multiple time scale series data; Based on the preset sliding window size and the time scale sequence data, respectively calculate the anomaly score of each time scale to obtain a plurality of first anomaly scores; Calculating the sensitivity of each time scale based on the first anomaly score and a preset score threshold, and calculating the abnormal data detection threshold based on the maximum value of the sensitivities; Based on the preset sliding window size, calculating the anomaly score of the to-be-detected data of the target device at the current moment to obtain a current anomaly score; Whether the data to be detected is abnormal data is determined according to a comparison result between the current abnormal score and the abnormal data detection threshold.

2. The device abnormal data detection method according to claim 1, characterized in that: The method of calculating the anomaly score of each time scale based on the preset sliding window size and the time scale sequence data to obtain a plurality of first anomaly scores includes: Calculate the moving average of each time scale based on the preset sliding window size and the time scale series data, and calculate the moving standard deviation of each time scale based on the moving average; The abnormality score of each time scale is calculated based on the time scale series data, the moving standard deviation and the moving average to obtain a plurality of first abnormality scores.

3. The device abnormal data detection method according to claim 2, characterized in that: The calculating the moving average of each time scale based on the preset sliding window size and the time scale sequence data respectively includes: respectively calculating the sum of the plurality of data in each of the time scale series data to obtain a first sum, and counting the number of the plurality of data in the time scale series data to obtain a first statistical result; The ratio of the first sum value to the first statistical result is calculated to obtain a first ratio, and the moving average value of each time scale is calculated based on a preset sliding window size and the first ratio by using a moving average method.

4. The device abnormal data detection method according to claim 3, characterized in that: The calculating of the anomaly score of each time scale based on the time scale series data, the moving standard deviation and the moving average to obtain a plurality of first anomaly scores includes: Calculate the difference between the first ratio and the moving average value corresponding to each time scale to obtain a target difference; The absolute value of the target difference is calculated, and the ratio of the absolute value to the moving standard deviation is calculated to obtain a first anomaly score corresponding to each time scale.

5. The device abnormal data detection method according to claim 1, characterized in that: The calculating the sensitivity of each time scale based on the first abnormality score and a preset score threshold includes: Obtaining scores greater than a preset score threshold from among the plurality of first anomaly scores corresponding to each time scale within the preset duration to obtain a second anomaly score, and counting the number of the second anomaly scores corresponding to each time scale to obtain a second statistical result; Calculating a sum of a plurality of the second anomaly scores corresponding to each time scale to obtain a second sum; The ratio of the second sum value to the second statistical result is calculated to obtain the sensitivity of the corresponding time scale.

6. The device abnormal data detection method according to claim 5, characterized in that: The calculating of the abnormal data detection threshold based on the maximum value among the sensitivities comprises: Determine a maximum value among the multiple sensitivities corresponding to the multiple time scales, and calculate a mean and a standard deviation of the first anomaly score corresponding to the maximum value within the preset time length to obtain a target mean and a target standard deviation; Determining a target time scale corresponding to the maximum value, and obtaining a preset adjustable parameter corresponding to the target time scale; The product of the target standard deviation and a preset adjustable parameter is calculated, and the sum of the product and the target mean is calculated to obtain an abnormal data detection threshold.

7. The device abnormal data detection method according to claim 6, characterized in that: The determining whether the data to be detected is abnormal data according to a comparison result between the current abnormal score and the abnormal data detection threshold comprises: If the current anomaly score is greater than the abnormal data detection threshold, the data to be detected is determined to be abnormal data; If the current abnormality score is not greater than the abnormal data detection threshold, the data to be detected is determined to be normal data.

8. The device abnormal data detection method according to claim 7, characterized in that: After determining whether the data to be detected is abnormal data according to the comparison result between the current abnormal score and the abnormal data detection threshold, the method further includes: If the data to be detected is abnormal data, the number of the data determined to be abnormal data within a preset time is counted to obtain a first number; Obtaining the number of false alarms in the abnormal data determined within the preset time to obtain a second number; Calculating a ratio of the second number to the first number to obtain a second ratio, and determining whether the second ratio is greater than a preset false alarm threshold; If the second ratio is greater than the preset false alarm threshold, the value of the preset adjustable parameter is reduced, and the step of extracting information from the historical time series data of the target device within a preset time length before the current moment according to different time scales to obtain multiple time scale series data is triggered.

9. The device abnormal data detection method according to claim 1, characterized in that: According to different time scales, information is extracted from the historical time series data of the target device within a preset time period before the current moment to obtain multiple time scale series data, including: When a preset time period is reached, historical operation data and / or historical status data of the components to be monitored in the server within a preset time period before the current moment are obtained to obtain historical time series data; the server is a server located in a preset data center; Information is extracted from the historical time series data according to the minute time scale, hour time scale, day time scale and week time scale to obtain time scale series data including minute level, hour level, day level and week level.

10. The device abnormal data detection method according to claim 9, characterized in that: The step of obtaining historical operation data and / or historical status data of the components to be monitored in the server within a preset time period before the current moment to obtain historical time series data includes: Get the CPU usage of the server recorded according to the preset collection period within the preset time before the current time, and obtain the historical time series data.

11. The device abnormal data detection method according to claim 9, characterized in that: After determining whether the data to be detected is abnormal data according to the comparison result between the current abnormal score and the abnormal data detection threshold, the method further includes: If the data to be detected is abnormal data, determining whether the data to be detected is greater than a preset usage rate threshold; If the data to be detected is greater than the preset usage rate threshold, a load abnormality alarm is triggered.

12. The device abnormal data detection method according to any one of claims 1 to 11, characterized in that: After determining whether the data to be detected is abnormal data according to the comparison result between the current abnormal score and the abnormal data detection threshold, the method further includes: If the data to be detected is abnormal data, alarm information including the abnormal data, the abnormal data detection threshold and the current abnormal score is generated so as to process the situation causing the abnormal data.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the device abnormal data detection method according to any one of claims 1 to 12 is implemented.

14. An electronic device, characterized in that: It comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the device abnormal data detection method as described in any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the device abnormal data detection method according to any one of claims 1 to 12 is implemented.

Images