Abnormality detection method and device, electronic equipment and storage medium

Abstract

The invention provides an anomaly detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining long time series data which comprises a plurality of first data points divided according to a preset initial time granularity and the cycle length information of the first data points; the period length information is adopted to convert the first data points of the long time sequence data, new time sequence data is obtained, and the new time sequence data comprises a plurality of second data points divided with the period length as the granularity; obtaining a target data point corresponding to the second data point according to a preset correlation condition, and obtaining a similarity value between the second data point and the corresponding target data point; when it is detected that the similarity value is smaller than a preset threshold value, it is determined that the second data point is abnormal, and whether the abnormity of the second data point meets the alarm noise reduction condition or not is judged; and if yes, performing alarm and noise reduction on the abnormity of the second data point. According to the embodiment of the invention, the processing efficiency and prediction result accuracy of intelligent operation and maintenance are improved.

Description

Technical Field

[0001] The present invention relates to the field of time series data anomaly detection, and in particular to an anomaly detection method, device, electronic equipment and storage medium. Background Art

[0002] With the continuous breakthroughs in software technology and the advancement of deep learning algorithms, AIOps (Artificial Intelligence for IT Operations) plays an increasingly important role in modern IT operations. AIOps can automatically identify problems in the system and accurately assign relevant events and information to the right people, thereby improving operation efficiency and response speed. However, although existing machine learning and deep learning models have achieved remarkable results in many fields, they still face challenges in processing massive data in intelligent operations.

[0003] After trying a variety of time series prediction algorithms, we found that existing tools have obvious limitations when processing long series data. For example, although the Prophet toolkit is suitable for predicting trends, its accuracy is not high; and although the autoregressive integrated moving average model algorithm performs well in short-sequence prediction, it is not accurate enough in trend prediction. In addition, although the classic long short-term memory network model performs well in long-sequence prediction, its serial structure leads to low operating efficiency, and when processing massive operation and maintenance data, the back propagation mechanism makes it difficult for the model to converge. These limitations make it difficult for existing algorithms to meet the efficiency and accuracy requirements of intelligent operation and maintenance when processing massive operation and maintenance data. Summary of the invention

[0004] In view of this, the present invention aims to propose an anomaly detection method, device, electronic device and storage medium to solve the problem that most of the current machine learning and deep learning models are not suitable for long sequence data. Even if there are models suitable for long sequence data, they will be inefficient during operation due to the serial structure, and the model is difficult to converge, resulting in low accuracy of prediction results. The specific technical solution is as follows:

[0005] According to a first aspect of the present invention, there is provided an anomaly detection method, the method comprising:

[0006] Acquire long time series data, wherein the long time series data includes a plurality of first data points divided according to a preset initial time granularity and period length information of the first data points;

[0007] The first data point of the long time series data is converted using the cycle length information to obtain new time series data, wherein the new time series data includes a plurality of second data points divided with the cycle length as a granularity;

[0008] Acquire a target data point corresponding to the second data point according to a preset correlation condition, and acquire a similarity value between the second data point and the corresponding target data point;

[0009] When it is detected that the similarity value is less than a preset threshold, determining that the second data point is abnormal, and judging whether the abnormality of the second data point satisfies the condition of alarm noise reduction;

[0010] If the conditions are met, the abnormality of the second data point is used for alarm noise reduction.

[0011] Optionally, the long time series data includes multiple indicator data;

[0012] After acquiring the long time series data, the method further includes:

[0013] Perform stability check and white noise check on various indicator data;

[0014] If it is detected that the stationarity check or the white noise check of the target indicator data among the multiple indicator data fails, the target indicator data is deleted from the long time series data to obtain updated multiple indicator data.

[0015] Optionally, the first data point of the long time series data carries a first timestamp;

[0016] The converting the first data point of the long time series data by using the cycle length information to obtain new time series data also includes:

[0017] Performing feature extraction on the updated multiple indicator data to obtain a number of feature information about the updated multiple indicator data;

[0018] The first data point of the long time series data is converted by using the plurality of feature information and the cycle length information to obtain new time series data including a plurality of second data points with the cycle length as a granularity, and the first timestamp is converted according to the cycle length information to obtain a second timestamp with the cycle length as a granularity;

[0019] A corresponding second timestamp is added to each second data point of the new time series data.

[0020] Optionally, acquiring a target data point corresponding to the second data point according to a preset correlation condition includes:

[0021] Acquire a target timestamp having a preset correlation with the second timestamp of each of the second data points;

[0022] A target data point corresponding to the second data point is acquired through the target timestamp.

[0023] Optionally, obtaining a similarity value between the second data point and the corresponding target data point includes:

[0024] According to different preset hash algorithms, multiple hash values ​​of each second data point and the target data point with correlation are obtained;

[0025] By presetting different weight values ​​for different hash algorithms, weighted summation is performed on multiple hash values ​​to obtain a similarity value between each second data point and a target data point with correlation, and half of the sum of the different weight values ​​is equal to the preset threshold.

[0026] Optionally, the determining whether the anomaly of the second data point satisfies a condition for alarm noise reduction further includes:

[0027] Predetermine the expected alarm duration for different anomaly types;

[0028] Obtain a target abnormality type of the second data point and an alarm duration of the target abnormality type;

[0029] Determining an expected alarm duration of the second data point according to the target anomaly type;

[0030] Determine whether the alarm duration is greater than or equal to the expected alarm duration.

[0031] Optionally, the alarm noise reduction includes alarm filtering, alarm merging or alarm compression;

[0032] After determining whether the alarm duration is greater than or equal to the expected alarm duration, the method further includes:

[0033] If the alarm duration is greater than or equal to the expected alarm duration, an abnormal alarm is sent;

[0034] If the alarm duration is shorter than the expected alarm duration, the anomaly of the second data point is subjected to alarm filtering, alarm merging or alarm compression.

[0035] According to a second aspect of the present invention, there is provided an abnormality detection device, the device comprising:

[0036] A first acquisition module is used to acquire long time series data, wherein the long time series data includes a plurality of first data points divided according to a preset initial time granularity and period length information of the first data points;

[0037] A conversion module, configured to convert the first data point of the long time series data using the cycle length information to obtain new time series data, wherein the new time series data includes a plurality of second data points divided with the cycle length as a granularity;

[0038] A second acquisition module, used to acquire a target data point corresponding to the second data point according to a preset correlation condition, and acquire a similarity value between the second data point and the corresponding target data point;

[0039] A judgment module, configured to determine that the second data point is abnormal when detecting that the similarity value is less than a preset threshold, and to judge whether the abnormality of the second data point satisfies a condition for alarm noise reduction;

[0040] The alarm noise reduction module is used to perform alarm noise reduction on the abnormality of the second data point if the condition is met.

[0041] According to another aspect of the present invention, there is also provided an electronic device, comprising:

[0042] processor;

[0043] a memory for storing instructions executable by the processor;

[0044] The processor is configured to execute the instructions to implement the anomaly detection method as described above.

[0045] According to another aspect of the present invention, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the anomaly detection method described above are implemented.

[0046] The anomaly detection method provided by the embodiment of the present invention obtains long time series data, wherein the long time series data includes a plurality of first data points divided according to a preset initial time granularity and period length information of the first data points; the first data points of the long time series data are converted using the period length information to obtain new time series data, wherein the new time series data includes a plurality of second data points divided according to the period length granularity; by changing the time granularity, the amount of data to be processed can be reduced and the processing efficiency can be improved; a target data point corresponding to the second data point is obtained according to a preset correlation condition, and a similarity value between the second data point and the corresponding target data point is obtained; when it is detected that the similarity value is less than a preset threshold, it is determined that an anomaly exists in the second data point, and it is determined whether the anomaly of the second data point meets the condition for alarm denoising; if so, the anomaly of the second data point is subjected to alarm denoising, and timely warning of abnormal situations can not only increase the predictability of the alarm, but also solve the fault in time to avoid the occurrence of larger anomalies. The embodiments of the present invention improve the processing efficiency of intelligent operation and maintenance by changing the time granularity of long time series data, reducing the amount of data to be processed, and at the same time avoiding erroneous operations by promptly handling abnormal situations and performing alarm noise reduction, thereby improving the stability of intelligent operation and maintenance and the accuracy of prediction results.

[0047] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0049] Figure 1 is a flowchart of a method for detecting anomalies provided by an embodiment of the present invention;

[0050] Figure 2 yes Figure 1 The flowchart of step 102 in an abnormality detection method provided by an embodiment of the present invention is shown;

[0051] Figure 3 yes Figure 1 A schematic diagram of time series data conversion in an anomaly detection method provided by an embodiment of the present invention is shown;

[0052] Figure 4 yes Figure 1 The flowchart of step 103 in an abnormality detection method provided by an embodiment of the present invention is shown;

[0053] Figure 5 is a structural schematic diagram of an abnormality detection device provided by an embodiment of the present invention;

[0054] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0056] Artificial Intelligence for IT Operations (AIOps). AIOps combines big data and machine learning to automate IT operation processes and implement functions such as event correlation, anomaly detection, and causal relationship determination. AIOps uses AI and its subsets to monitor and solve problems related to enterprise IT operations. An intelligent operation scenario can refer to a customer's normal business involving multiple operation indicator data. Through intelligent operation technology, multiple indicators are jointly analyzed to determine whether the business data is abnormal, reduce the workload of operation personnel, and improve their work efficiency. However, the more mature models of machine learning and deep learning at this stage are only suitable for short-term predictions, and are not suitable for tasks in intelligent operation that respond to massive amounts of data at the minute level. Common supervised classification learning algorithms that use machine learning solutions in intelligent operation require more learning costs and are difficult to solve the problem of extremely unbalanced positive and negative samples in the task itself; similarly, unsupervised learning clustering methods cannot quickly analyze complex business scenarios with multiple operation indicators. Deep learning algorithms popular in recent years, such as LSTM algorithm and Transformer algorithm, cannot solve the problems of difficulty in model convergence on massive operation and maintenance data, high computational cost, and complex deployment. Therefore, the present invention proposes an anomaly detection method.

[0057] Reference Figure 1 , shows a flowchart of the steps of an abnormality detection method provided by an embodiment of the present invention, the method may include:

[0058] Step 101: Acquire long time series data, where the long time series data includes a number of first data points divided according to a preset initial time granularity and period length information of the first data points.

[0059] The long time series data in the embodiment of the present invention includes a variety of indicator data. Before analyzing the long time series data, a series of preprocessing operations are usually required to ensure the accuracy and availability of the data. These preprocessing operations include filling empty data, cleaning dirty data, etc. Filling empty data means that in the actual data collection process, some data points may be missing due to various reasons. Therefore, the mean, median, interpolation method, etc. are used to fill the empty data to ensure the continuity and integrity of the data. Dirty data refers to data that does not conform to the expected format or contains outliers. Methods for cleaning dirty data include deleting outliers, correcting erroneous data, etc. to ensure the quality of the data.

[0060] After preprocessing, the stationarity and randomness of individual indicator data in the long time series data will also be tested. The specific steps include:

[0061] Perform stability check and white noise check on various indicator data;

[0062] If it is detected that the stationarity check or white noise check of the target indicator data among the multiple indicator data fails, the target indicator data is deleted from the long time series data to obtain updated multiple indicator data.

[0063] Among them, the stationarity test can be ADF (Augmented Dickey-Fuller test), which can determine whether the long time series data is stationary. The test is only for one indicator data at a time. When ADF is tested, it is necessary to first construct an ADF statistic through a certain indicator data of the long time series data, and then perform a hypothesis test through the ADF statistic and the critical value. The critical value is usually obtained from the Dickey-Fuller distribution table. If the ADF statistic is less than the critical value, the original hypothesis is rejected, and it is believed that this indicator data is stationary, and the next step of analysis can be carried out. If the ADF statistic is greater than the critical value, the original hypothesis is not rejected, and it is believed that this indicator data is non-stationary. At this time, the data needs to be deleted from the long time series data.

[0064] The main purpose of white noise verification for long time series data is to determine whether the long time series data is purely random. If the long time series data is white noise data, it means that it is a purely random change, corresponding to a random variable that does not contain any trend, seasonality or periodic components. Such data cannot be predicted by fitting functions, so the indicator data is removed and not analyzed. The white noise verification steps include: selecting an indicator data from the long time series data, calculating the autocorrelation function (ACF) of this indicator data, observing its correlation, and judging whether the time series data is white noise through ACF graphs and statistical tests (such as Ljung-Box test).

[0065] After filling empty data and cleaning dirty data, the data quality and availability can be ensured by testing the stability and white noise of individual indicator data. The ADF stability test is used to determine whether the long time series data is stable. If it is not stable, the indicator data is removed; the ACF white noise test is used to determine whether the long time series data is white noise. If it is white noise, the indicator data is removed. These steps help improve the accuracy and reliability of data analysis and provide a solid foundation for subsequent modeling and prediction.

[0066] It should be noted that the long time series data is composed of the first data points divided according to the preset initial time granularity. These first data points will also be updated according to the preset period, so the acquired long time series data also includes the period length information of the first data points. The long time series data includes multiple indicator data, so that when detecting anomalies, the operation and maintenance personnel can conduct a comprehensive analysis of the multi-indicator data to make the obtained anomaly detection more accurate.

[0067] Step 102, using the period length information to transform the first data point of the long time series data to obtain new time series data, where the new time series data includes a plurality of second data points divided with the period length as the granularity.

[0068] After obtaining the long time series data in the embodiment of the present invention, the long time series data is first converted into multi-indicator panel data according to the indicator information and initial time granularity contained in the long time series data. This data is similar to the structure of image data that deep learning is good at solving. Then the multi-indicator panel data is checked for stationarity and white noise, and after deleting the non-compliant indicator data, the remaining long time series data is experimentally expanded to obtain several feature information about the updated multiple indicator data. Among them, statistical features can be obtained during feature extraction: such as mean, variance, maximum value, minimum value, median, etc., time domain features: such as autocorrelation function (ACF), partial autocorrelation function (PACF), spectral density, etc., frequency domain features: such as Fourier transform, wavelet transform, etc., time-frequency features: such as short-time Fourier transform (STFT), wavelet packet transform, etc. Then, according to the preset cycle length (such as daily, weekly, monthly, hourly, etc., the present invention does not make specific limitations here), the long time series data is grouped, and the data in each cycle is aggregated (such as sum, average, maximum value, minimum value, etc.), and then the extracted feature information and the aggregated data volume and cycle length are combined to obtain a new second data point with the structure of panel data. It should be noted that the first data point of the long time series data carries the first timestamp, so at the same time, according to the cycle length information, the original first timestamp is converted into a second timestamp with the cycle length as the granularity.

[0069] Further, step 102, such as Figure 2 As shown:

[0070] Step 1021 , extracting features from the updated multiple indicator data to obtain some feature information about the updated multiple indicator data.

[0071] Step 1022, use some characteristic information and cycle length information to transform the first data point of the long time series data to obtain new time series data containing some second data points with cycle length as granularity, and at the same time, transform the first timestamp according to the cycle length information to obtain the second timestamp with cycle length as granularity.

[0072] Step 1023: Add a corresponding second timestamp to each second data point of the new time series data.

[0073] For example, Figure 3As shown, if there are 8 kinds of indicator data in the updated multiple indicator data, the initial time granularity is minute granularity, the cycle length is day, the first timestamps are 2023-10-01 20:17, 2023-10-01 20:18, 2023-10-01 20:19, 2023-10-01 20:20, and are expanded to 32 feature information through experiments. The panel data is regarded as data032*n, and it is converted to obtain the second data point. The panel data can be regarded as data1 32*1440, where 1440=24×60, and the corresponding second timestamp is 2023-10-01, where data0 is the first timestamp and data1 is the second timestamp. If the cycle length is two hours, the panel data converted according to the cycle length can be regarded as data2 32*120, 120=60×2, and data2 is also the converted timestamp, and the corresponding converted timestamp is 20:00 on October 1, 2023. Of course, other time intervals can also be converted for the corresponding timestamp, and the present invention is not specifically limited here.

[0074] By performing feature extraction, data conversion, and timestamp addition on the updated multiple indicator data to generate new time series data, and obtaining the target data points related to the second data points and their similarity values, it helps to improve the accuracy and reliability of data analysis and provide a solid foundation for subsequent modeling and prediction.

[0075] Step 103: acquiring a target data point corresponding to the second data point according to a preset correlation condition, and acquiring a similarity value between the second data point and the corresponding target data point.

[0076] In the embodiment of the present invention, the preset correlation condition can be set according to the time window or according to the data correlation. When acquiring, the timestamp is first acquired, and then the selected timestamp is matched with the target data point to ensure that each target timestamp corresponds to a target data point. The preset different hash algorithms include but are not limited to the mean hash algorithm, the difference hash algorithm and the perceptual hash algorithm. Different weight values ​​are set for each hash algorithm to ensure that half of the sum of different weight values ​​is equal to the preset threshold, and then the hash algorithm is used to perform hash calculation on each second data point and the target data point with correlation to obtain multiple hash values, and the multiple hash values ​​are weighted and summed to obtain the final similarity value.

[0077] Further, step 103, such as Figure 4 As shown:

[0078] Step 1031: Obtain a target timestamp that has a preset correlation with the second timestamp of each second data point.

[0079] Step 1032: Obtain a target data point corresponding to the second data point through the target timestamp.

[0080] Step 1033: Obtain multiple hash values ​​of each second data point and the target data point with which there is correlation according to different preset hash algorithms.

[0081] Step 1034, by pre-setting different weight values ​​for different hash algorithms, weighted summing is performed on multiple hash values ​​to obtain a similarity value between each second data point and the target data point with which there is correlation, and half of the sum of the different weight values ​​is equal to the preset threshold.

[0082] For example, if the second timestamp of the second data point is in days, the preset correlation condition is to obtain the target timestamps of the same period yesterday, the day before yesterday and last week of the second data point to be detected (the second timestamp is 2024-11-18), which are 2024-11-17, 2024-11-16 and 2024-11-11 respectively. If the initial time granularity is minute granularity, when detecting whether an abnormality occurs in a certain minute of data, because the minute granularity is small, the present application can expand the comparison range and set the focus on the data a few hours before that moment. For example, the second timestamp of the second data point to be detected is 2024-11-1820:19, and the target timestamps related to it are obtained: 2024-11-1818:00, 2024-11-18 16:00 and 2024-11-1120:00 (that is, the time value range is two hours). After obtaining the target timestamp, obtain the target data point corresponding to the target timestamp, calculate the hash values ​​to be 1, 0, 1, set the hash algorithms to be mean hash algorithm, difference hash algorithm and perceptual hash algorithm, and the corresponding weights are 0.5, 0.4 and 0.5 respectively. At this time, the similarity value = 0.5×1+0.4×0+0.5×1=1, and the preset threshold = (0.5+0.4+0.5)÷2=0.7.

[0083] It should be noted that different weight values ​​set in advance for different hash algorithms can be dynamically adjusted according to needs and actual conditions, so that the preset thresholds will also be adjusted accordingly, so that the thresholds can be changed as needed for various situations such as data of different magnitudes and data that fluctuates greatly and irregularly, thereby improving the flexibility of the solution and avoiding over-reliance on the professionalism of operation and maintenance personnel.

[0084] Step 104, when it is detected that the similarity value is less than the preset threshold, it is determined that the second data point has an anomaly, and it is determined whether the anomaly of the second data point meets the condition of alarm noise reduction.

[0085] The alarm noise reduction in the embodiment of the present invention includes alarm filtering, alarm merging or alarm compression. The preset threshold is determined by setting different weight values ​​for different hash algorithms, and can be adjusted as needed to avoid over-reliance on operation and maintenance personnel, reduce labor costs, and enhance the flexibility of threshold setting.

[0086] After obtaining the similarity value, the similarity value is compared with the preset threshold. If the similarity value is low, it is considered that the second data point at this time has an abnormality. However, not all alarms are useful or need to be processed immediately. Some alarms may be repeated, irrelevant, false alarms or minor. Therefore, the embodiment of the present invention filters, merges or compresses alarms according to business characteristics, making it easier for operators to identify and process important alarms, improve operating efficiency and system stability. For example, because the business scenario is monitored in real time, once an abnormality occurs, alarms may be generated for a period of time, resulting in alarm redundancy, so alarms are continuously merged or compressed for this type of abnormality. When making a judgment, first determine that the expected alarm duration required for different abnormal types is different. For example, according to the severity and impact range of the abnormality, the abnormal type is classified, such as severe abnormality, medium abnormality, slight abnormality, etc., and an expected alarm duration is set for each abnormal type. For example, a severe abnormality may alarm within 1 minute, a medium abnormality may alarm within 5 minutes, and a slight abnormality may alarm within 10 minutes. Then, the alarm duration of the target abnormality type to which the second data point belongs is obtained, and according to the detected target abnormality type, a preset expected alarm duration is found, and according to the alarm duration and the expected alarm duration, whether alarm noise reduction is required is determined. The specific steps include:

[0087] Predetermine the expected alarm duration for different anomaly types;

[0088] Obtain the target abnormality type and the alarm duration of the target abnormality type of the second data point;

[0089] Determine the expected alarm duration of the second data point according to the target anomaly type;

[0090] Determine whether the alarm duration is greater than or equal to the expected alarm duration.

[0091] If the alarm duration is greater than or equal to the expected alarm duration, an abnormal alarm is sent;

[0092] If the alarm duration is shorter than the expected alarm duration, the anomaly of the second data point is subjected to alarm filtering, alarm merging or alarm compression.

[0093] For example, suppose the second data point anomaly detected is that the CPU usage of the server exceeds 90%, which is a serious anomaly. The alarm duration is 30 seconds, and the expected alarm duration is 1 minute. Since the alarm duration is less than the expected alarm duration, no alarm is sent at this time, but the alarm is filtered, merged or compressed. Suppose the second data point anomaly detected is that the memory usage exceeds 80%, which is a medium anomaly. The alarm duration is 6 minutes, and the expected alarm duration is 5 minutes. Since the alarm duration is longer than the expected alarm duration, the alarm notification needs to be resent at this time to prompt the staff to handle it as soon as possible.

[0094] By setting the expected alarm duration for different types of anomalies and judging and processing according to the alarm duration in actual operation, the accuracy and efficiency of alarms can be effectively improved. Alarm filtering, merging and compression can help reduce redundant alarms and improve the work efficiency of operation and maintenance personnel.

[0095] It should be noted that in addition to the above-mentioned methods, other methods can also be used to reduce alarm noise. For example, if the anomaly detected in the second data point is a regular, repetitive large fluctuation, then this anomaly will also be filtered.

[0096] Step 105: If the condition is satisfied, the abnormality of the second data point is subjected to alarm denoising.

[0097] In the embodiment of the present invention, when it is determined that the anomaly of the second data point meets the condition of alarm noise reduction, alarm noise reduction will be performed. Through alarm noise reduction, business personnel can determine the real anomaly, avoid misjudgment or repeated processing of some special cases, and improve processing efficiency.

[0098] The anomaly detection method provided by the embodiment of the present invention obtains long time series data, which includes a plurality of first data points divided according to a preset initial time granularity and the period length information of the first data points; the first data point of the long time series data is converted by using the period length information to obtain new time series data, and the new time series data includes a plurality of second data points divided according to the period length granularity; by changing the time granularity, the amount of data processed can be reduced and the processing efficiency can be improved; the target data point corresponding to the second data point is obtained according to the preset correlation condition, and the similarity value between the second data point and the corresponding target data point is obtained; when the similarity value is detected to be less than the preset threshold, it is determined that the second data point has an anomaly, and it is determined whether the anomaly of the second data point meets the condition of alarm denoising; if it meets the condition, the anomaly of the second data point is alarm denoised, and through the timely warning of the abnormal situation, not only the predictability of the alarm can be increased, but also the fault can be solved in time to avoid the occurrence of a larger anomaly. The embodiment of the present invention reduces the amount of data processed and improves the processing efficiency by changing the time granularity of the long time series data, and at the same time, by timely processing the abnormal situation and performing alarm denoising, erroneous operations are avoided, thereby improving the stability of the system and the accuracy of the prediction results.

[0099] Reference Figure 5 , shows a schematic diagram of the structure of an abnormality detection device provided by an embodiment of the present invention, the device comprising:

[0100] The first acquisition module 201 is used to acquire long time series data, where the long time series data includes a number of first data points divided according to a preset initial time granularity and period length information of the first data points.

[0101] The conversion module 202 is used to convert the first data point of the long time series data using the period length information to obtain new time series data, where the new time series data includes a plurality of second data points divided with the period length as the granularity.

[0102] The second acquisition module 203 is used to acquire the target data point corresponding to the second data point according to a preset correlation condition, and acquire the similarity value between the second data point and the corresponding target data point.

[0103] The judgment module 204 is used to determine whether the second data point has an abnormality when detecting that the similarity value is less than a preset threshold, and to judge whether the abnormality of the second data point meets the condition of alarm noise reduction.

[0104] The alarm noise reduction module 205 is used to perform alarm noise reduction on the abnormality of the second data point if the condition is met.

[0105] Optionally, the long time series data includes multiple indicator data.

[0106] The anomaly detection device also includes:

[0107] The verification module is used to perform stationarity verification and white noise verification on various indicator data.

[0108] The deletion module is used to delete the target indicator data from the long time series data if it is detected that the stationarity check or white noise check of the target indicator data in the multiple indicator data fails, so as to obtain the updated multiple indicator data.

[0109] Optionally, the first data point of the long time series data carries a first timestamp.

[0110] The conversion module 202 specifically includes:

[0111] The feature extraction submodule is used to extract features from the updated multiple indicator data to obtain a number of feature information about the updated multiple indicator data.

[0112] The conversion submodule is used to convert the first data point of the long time series data using a number of feature information and cycle length information to obtain new time series data containing a number of second data points with the cycle length as the granularity, and at the same time convert the first timestamp according to the cycle length information to obtain the second timestamp with the cycle length as the granularity.

[0113] The timestamp adding submodule is used to add a corresponding second timestamp to each second data point of the new time series data.

[0114] Optionally, the second acquisition module 203 specifically includes:

[0115] The first acquisition submodule is used to acquire a target timestamp that has a preset correlation with the second timestamp of each second data point.

[0116] The second acquisition submodule is used to acquire a target data point corresponding to the second data point through a target timestamp.

[0117] The third acquisition submodule is used to obtain multiple hash values ​​of each second data point and the target data point with correlation according to different preset hash algorithms.

[0118] The similarity acquisition submodule is used to set different weight values ​​for different hash algorithms in advance, perform weighted summation on multiple hash values, and obtain the similarity value of each second data point with the target data point with correlation, and half of the sum of the different weight values ​​is equal to the preset threshold.

[0119] Optionally, the determination module 204 specifically includes:

[0120] The first determination submodule is used to predetermine the expected alarm durations of different abnormality types.

[0121] The fourth acquisition submodule is used to obtain the target abnormality type of the second data point and the alarm duration of the target abnormality type.

[0122] The second determination submodule is used to determine the expected alarm duration of the second data point according to the target abnormality type.

[0123] The judgment submodule is used to judge whether the alarm duration is greater than or equal to the expected alarm duration.

[0124] Optionally, alarm noise reduction includes alarm filtering, alarm merging or alarm compression.

[0125] The alarm sending submodule is used to send an abnormal alarm if the alarm duration is greater than or equal to the expected alarm duration.

[0126] The alarm noise reduction submodule is used to filter, merge or compress the abnormality of the second data point if the alarm duration is shorter than the expected alarm duration.

[0127] The anomaly detection method provided by the embodiment of the present invention obtains long time series data, which includes a plurality of first data points divided according to a preset initial time granularity and the period length information of the first data points; the first data point of the long time series data is converted by using the period length information to obtain new time series data, and the new time series data includes a plurality of second data points divided according to the period length granularity; by changing the time granularity, the amount of data processed can be reduced and the processing efficiency can be improved; the target data point corresponding to the second data point is obtained according to the preset correlation condition, and the similarity value between the second data point and the corresponding target data point is obtained; when the similarity value is detected to be less than the preset threshold, it is determined that the second data point has an anomaly, and it is determined whether the anomaly of the second data point meets the condition of alarm denoising; if it meets the condition, the anomaly of the second data point is alarm denoised, and through the timely warning of the abnormal situation, not only the predictability of the alarm can be increased, but also the fault can be solved in time to avoid the occurrence of a larger anomaly. The embodiment of the present invention reduces the amount of data processed and improves the processing efficiency by changing the time granularity of the long time series data, and at the same time, by timely processing the abnormal situation and performing alarm denoising, erroneous operations are avoided, thereby improving the stability of the system and the accuracy of the prediction results.

[0128] Reference Figure 6 , shows an electronic device provided by an embodiment of the present invention, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement any anomaly detection method.

[0129] Figure 61 is a block diagram of an electronic device 300 according to an exemplary embodiment. For example, the electronic device 300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0130] Reference Figure 6 , the electronic device 300 may include one or more of the following components: a processing component 302 , a memory 304 , a power component 306 , a multimedia component 308 , an audio component 310 , an input / output interface 312 , a sensor component 314 , and a communication component 316 .

[0131] The processing component 302 generally controls the overall operation of the device 300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 302 may include one or more modules to facilitate the interaction between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate the interaction between the multimedia component 308 and the processing component 302.

[0132] The memory 304 is configured to store various types of data to support operations on the device 300. Examples of such data include instructions for any application or method operating on the device 300, contact data, phone book data, messages, pictures, videos, etc. The memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0133] The power supply component 306 provides power to the various components of the electronic device 300. The power supply component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 300.

[0134] The multimedia component 308 includes a screen that provides an output interface between the electronic device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 includes a front camera and / or a rear camera. When the electronic device 300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0135] The audio component 310 is configured to output and / or input audio signals. For example, the audio component 310 includes a microphone (MIC), and when the electronic device 300 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 304 or sent via the communication component 316. In some embodiments, the audio component 310 also includes a speaker for outputting audio signals.

[0136] The input / output interface 312 provides an interface between the processing component 302 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0137] The sensor assembly 314 includes one or more sensors for providing various aspects of status assessment for the electronic device 300. For example, the sensor assembly 314 can detect the open / closed state of the electronic device 300, the relative positioning of components, such as the display and keypad of the electronic device 300, and the sensor assembly 314 can also detect the position change of the electronic device 300 or a component of the electronic device 300, the presence or absence of user contact with the electronic device 300, the orientation or acceleration / deceleration of the electronic device 300, and the temperature change of the electronic device 300. The sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 314 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0138] The communication component 316 is configured to facilitate wired or wireless communication between the electronic device 300 and other devices. The electronic device 300 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0139] In an exemplary embodiment, the electronic device 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0140] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, and the instructions can be executed by a processor 320 of an electronic device 300 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0141] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0142] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0143] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An anomaly detection method, characterized in that: The method comprises: Acquire long time series data, wherein the long time series data includes a plurality of first data points divided according to a preset initial time granularity and period length information of the first data points; The first data point of the long time series data is converted using the cycle length information to obtain new time series data, wherein the new time series data includes a plurality of second data points divided with the cycle length as a granularity; Acquire a target data point corresponding to the second data point according to a preset correlation condition, and acquire a similarity value between the second data point and the corresponding target data point; When it is detected that the similarity value is less than a preset threshold, determining that the second data point is abnormal, and judging whether the abnormality of the second data point satisfies the condition of alarm noise reduction; If the conditions are met, the abnormality of the second data point is used for alarm noise reduction.

2. The method according to claim 1, characterized in that: The long time series data includes a variety of indicator data; After acquiring the long time series data, the method further includes: Perform stability check and white noise check on various indicator data; If it is detected that the stationarity check or the white noise check of the target indicator data among the multiple indicator data fails, the target indicator data is deleted from the long time series data to obtain updated multiple indicator data.

3. The method according to claim 2, characterized in that: The first data point of the long time series data carries a first timestamp; The converting the first data point of the long time series data by using the cycle length information to obtain new time series data also includes: Performing feature extraction on the updated multiple indicator data to obtain a number of feature information about the updated multiple indicator data; The first data point of the long time series data is converted by using the plurality of feature information and the cycle length information to obtain new time series data including a plurality of second data points with the cycle length as a granularity, and the first timestamp is converted according to the cycle length information to obtain a second timestamp with the cycle length as a granularity; A corresponding second timestamp is added to each second data point of the new time series data.

4. The method according to claim 3, characterized in that: The acquiring the target data point corresponding to the second data point according to the preset correlation condition includes: Acquire a target timestamp having a preset correlation with the second timestamp of each of the second data points; A target data point corresponding to the second data point is acquired through the target timestamp.

5. The method according to claim 1, characterized in that: The obtaining a similarity value between the second data point and the corresponding target data point includes: According to different preset hash algorithms, multiple hash values ​​of each second data point and the target data point with correlation are obtained; By presetting different weight values ​​for different hash algorithms, weighted summation is performed on multiple hash values ​​to obtain a similarity value between each second data point and a target data point with correlation, and half of the sum of the different weight values ​​is equal to the preset threshold.

6. The method according to claim 1, characterized in that: The determining whether the abnormality of the second data point satisfies the condition of alarm noise reduction further includes: Predetermine the expected alarm duration for different anomaly types; Obtain a target abnormality type of the second data point and an alarm duration of the target abnormality type; Determining an expected alarm duration of the second data point according to the target anomaly type; Determine whether the alarm duration is greater than or equal to the expected alarm duration.

7. The method according to claim 6, characterized in that: The alarm noise reduction includes alarm filtering, alarm merging or alarm compression; After determining whether the alarm duration is greater than or equal to the expected alarm duration, the method further includes: If the alarm duration is greater than or equal to the expected alarm duration, an abnormal alarm is sent; If the alarm duration is shorter than the expected alarm duration, the anomaly of the second data point is subjected to alarm filtering, alarm merging or alarm compression.

8. An abnormality detection device, characterized in that: The device comprises: A first acquisition module is used to acquire long time series data, wherein the long time series data includes a plurality of first data points divided according to a preset initial time granularity and period length information of the first data points; A conversion module, configured to convert the first data point of the long time series data using the cycle length information to obtain new time series data, wherein the new time series data includes a plurality of second data points divided with the cycle length as a granularity; A second acquisition module, used to acquire a target data point corresponding to the second data point according to a preset correlation condition, and acquire a similarity value between the second data point and the corresponding target data point; A judgment module, configured to determine that the second data point is abnormal when detecting that the similarity value is less than a preset threshold, and to judge whether the abnormality of the second data point satisfies a condition for alarm noise reduction; The alarm noise reduction module is used to perform alarm noise reduction on the abnormality of the second data point if the condition is met.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the abnormality detection method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the abnormality detection method according to any one of claims 1 to 7 is implemented.

Images