Time sequence anomaly detection and evaluation method and system based on operation and maintenance attention

Abstract

The invention discloses a time sequence anomaly detection and evaluation method and system based on operation and maintenance attention, and the method comprises the steps: obtaining a time sequence of a performance index in a period of time and a corresponding ground true phase label, carrying out the anomaly detection of the time sequence through an anomaly detection model, and outputting an anomaly detection result; calculating the operation and maintenance attention of each abnormal point in the ground true phase label and the abnormal detection result one by one, and predicting the operation and maintenance attention of all time points in the next observation period length according to the operation and maintenance attention of the abnormal point, thereby obtaining an operation and maintenance attention curve of the ground true phase label and the abnormal detection result; and evaluating the anomaly detection result of the anomaly detection model. The invention relates to the field of infrastructure and I T support calculation, can consider the continuity of an abnormal event in time sequence when the output of a time sequence anomaly detection model is evaluated, avoids the defect of overestimation of long abnormal events, and provides a reasonable evaluation result for a fragmented anomaly detection result.

Description

Technical Field

[0001] The present invention relates to a time series anomaly detection and evaluation method and system based on operation and maintenance attention, and relates to the fields of infrastructure and IT support, artificial intelligence, cloud computing, big data and edge computing. Background Art

[0002] In today's rapidly developing mobile Internet era, with the expansion of system scale and diversification of business, traditional operation and maintenance methods have been unable to cope with the growing monitoring needs. Therefore, operation and maintenance personnel automatically monitor key performance indicators (KPIs) related to system stability, such as business volume and success rate, to effectively ensure the smooth operation of the system and improve user experience. In this case, time series anomaly detection models play an important role. They can monitor and analyze a large amount of KPI data in real time, and promptly discover and warn of potential system anomalies, thereby helping the operation and maintenance team to respond quickly and reduce the impact of system failures on the business.

[0003] Evaluation methods for time series anomaly detection models are critical to ensuring the effectiveness and accuracy of these models. Evaluation methods can help technicians understand how the model performs in real-world applications, including its ability to identify anomalies, false positive and false negative rates, etc. Through these evaluations, the model can be optimized and adjusted to better adapt to changing business needs and system environments.

[0004] In the existing known technologies, the basic binary classification model uses the traditional precision, recall and F1 score indicators to evaluate its classification performance. Among them, precision represents the ratio of the number of correctly detected anomalies to all points detected as anomalies, recall represents the ratio of the number of correctly detected anomalies to all true anomalies, and F1 score is the harmonic mean of precision and recall. In these basic methods, the calculation of indicators is based on points, each sample point is considered independent, and the weight of each sample point is considered equal. This method does not take into account the continuous nature of time series in time series anomaly detection applications. For example, a single abnormal sample point and several abnormal sample points that are continuous in time series can be described by "one abnormal event", so this basic method is not suitable for the evaluation of time series anomaly detection models.

[0005] In recent years, many time series anomaly detection studies have used the point adjustment (PA) method to correct the output labels before performing F1 scoring. The working principle of PA is as follows: if at least one point in a continuous abnormal event is detected as an anomaly, the entire abnormal event is considered to be correctly detected as an anomaly. This method embodies a certain idea of ​​taking abnormal events as units. However, in essence, it still uses points as evaluation calculation units. This method has the phenomenon of overestimation of the model. This is because some abnormal events last for a long time. If one abnormal point is detected in these abnormal events, it is considered that the entire event is detected. Then the weight of the event in the evaluation result is too high, which will cause other abnormal events with shorter duration to become insignificant. At this point, it violates the idea of ​​being based on abnormal events and leads to unreasonable evaluation.

[0006] The existing patent CN202211312131.4 (application name: A time series anomaly detection and evaluation method and system based on abnormal segments, applicant: Institute of Information Engineering, Chinese Academy of Sciences, application date: 2022.10.25) discloses a time series anomaly detection and evaluation method and system based on abnormal segments. This technical solution uses abnormal segments as evaluation calculation units instead of points. For the calculation of precision and recall, this technical solution calculates in units of abnormal segments and takes the average value between each segment. This technical solution uses a weight curve to highlight that points at different positions in the abnormal segment have different weight scores. However, this technical solution has the following problems: On the one hand, the abnormal segment merging method in this method cannot handle fragmented detection results. Due to the discrete sampling of the time series, the anomaly detection model outputs discrete detection results, and there may be a small number of FP (false positive) sample points mixed in with a large number of TP (true positive) sample points, but they belong to the same abnormal event. On the other hand, this method considers abnormal segments of different lengths to be completely equal, but in fact, the importance of abnormal events with very long duration (e.g., more than 8 hours) and those with very short duration (e.g., 1 minute) is very different. This is different from the fact that the importance of short-duration abnormalities (1 minute and 2 minutes) is basically similar. This method does not take this issue into account, and may therefore lead to an underestimation of the importance of abnormal events with longer duration.

[0007] Therefore, how to fully consider the continuity of abnormal events in the time series in the temporal order when evaluating the output results of the time series anomaly detection model, effectively avoid the defect of overestimating the performance of long abnormal events, and at the same time give reasonable evaluation results in the case of fragmented anomaly detection result output has become a technical issue that technical personnel have focused on. Summary of the invention

[0008] In view of this, the purpose of the present invention is to provide a time series anomaly detection evaluation method and system based on operation and maintenance attention. When evaluating the output results of the time series anomaly detection model, it can not only fully consider the continuity of abnormal events in the time series in time sequence, but also effectively avoid the defect of overestimating the performance of long abnormal events. At the same time, it can also give reasonable evaluation results in the case of fragmented anomaly detection result output.

[0009] In order to achieve the above object, the present invention provides a time series anomaly detection and evaluation method based on operation and maintenance attention, comprising:

[0010] Step 1: Obtain the time series of performance indicators over a period of time X = {x0, x1, ..., x T-1} and the corresponding ground truth labels Y = {y0, y1, ..., y T-1}, T is the total number of time points in the time series X, y t ∈{0,1},t∈{0,1,…,T-1},y t = 0 or 1 indicates that the state of the time series at time point t is normal or abnormal. Then the anomaly detection model is used to detect anomalies in the time series X and output the anomaly detection results. or 1 indicates that the anomaly detection result of the anomaly detection model at time point t is normal or abnormal;

[0011] Step 2: Calculate the ground truth labels Y and anomaly detection results one by one The operation attention of each anomaly point in is used to predict the operation attention of all time points within the next observation period, thereby obtaining the ground truth label Y and the anomaly detection result. The operation and maintenance attention curve is used to evaluate the anomaly detection results of the anomaly detection model.

[0012] In order to achieve the above object, the present invention also provides a time series anomaly detection and evaluation system based on operation and maintenance attention, comprising:

[0013] A time series acquisition device is used to obtain the time series X = {x0, x1, ..., xT-1} of the performance indicators within a period of time and the corresponding ground truth labels Y = {y0, y1, ..., y T-1}, T is the total number of time points in the time series X, y t ∈{0,1},t∈{0,1,…,T-1},y t = 0 or 1 indicates that the state of the time series at time point t is normal or abnormal. Then the anomaly detection model is used to detect anomalies in the time series X and output the anomaly detection results. or 1 means that the anomaly detection result of the anomaly detection model at time point t is normal or abnormal;

[0014] Anomaly detection evaluation device, which is used to calculate the ground truth label Y and anomaly detection results one by one The operation attention of each anomaly point in is used to predict the operation attention of all time points within the next observation period, thereby obtaining the ground truth label Y and the anomaly detection result. The operation and maintenance attention curve is used to evaluate the anomaly detection results of the anomaly detection model.

[0015] In order to achieve the above object, the present invention further provides a computing device, comprising:

[0016] Memory and processor;

[0017] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the time series anomaly detection and evaluation method based on operation and maintenance attention are implemented.

[0018] In order to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the time series anomaly detection and evaluation method based on operation and maintenance attention are implemented.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: by defining the operation and maintenance attention curve of time series anomaly detection, the present invention can establish a behavioral model for operation and maintenance personnel to monitor the output of anomaly detection models and handle abnormal alarms during actual operation and maintenance, and visualize it. The use of the present invention to evaluate the performance of anomaly detection models is in line with the cognition and work habits of operation and maintenance personnel, and enhances the interpretability and practical effect of the evaluation method. The present invention also proposes an online calculation method for the operation and maintenance attention curve, which uses a front-to-backward method to calculate the operation and maintenance attention curves of continuous abnormal events and fragmented abnormal events, and can merge fragmented detection results, effectively evaluate the discretized output of the general anomaly detection model, and at the same time, it is not necessary to know the future ground truth labels and anomaly detection model outputs, so it is suitable for operation and maintenance attention calculation in online scenarios. ; The present invention also proposes a method for calculating precision and recall based on the operation and maintenance attention curve. The method converts the calculation of the number of true positive samples, false positive samples and false negative samples in the traditional method into the calculation of the overlapping area and the area under the curve of the operation and maintenance attention curve, and proposes a new evaluation perspective for time series anomaly detection models, which is different from the traditional point-based evaluation method and some event-based evaluation methods, and avoids the evaluation defects of these two methods in specific situations; The present invention also proposes a parameter setting method for time series anomaly detection evaluation based on operation and maintenance attention, including an empirical method and an adaptive method, which does not require complex manual parameter setting and configuration, and therefore has technical simplicity, which can help operation and maintenance personnel to simply use the time series anomaly detection evaluation method based on operation and maintenance attention to evaluate, sort and select anomaly detection models. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of an abnormal event and its operation and maintenance attention curve shown in an exemplary embodiment of the present invention.

[0021] Figure 2 It is a flowchart of a time series anomaly detection and evaluation method based on operation and maintenance attention shown in an exemplary embodiment of the present invention.

[0022] Figure 3 It is a schematic diagram of an operation and maintenance attention curve of a fragmented detection result shown in an exemplary embodiment of the present invention.

[0023] Figure 4 An exemplary embodiment of the present invention is shown Figure 2 In step 2, a flowchart of the specific process of evaluating the anomaly detection results of the anomaly detection model.

[0024] Figure 5 is a TP calculated based on the area under the operation and maintenance attention curve shown in an exemplary embodiment of the present invention. oi , FPoi and FN oi Schematic diagram of .

[0025] Figure 6 It is a structural diagram of a time series anomaly detection and evaluation system based on operation and maintenance attention shown in an exemplary embodiment of the present invention.

[0026] Figure 7 It is a schematic diagram of the structure of a computer device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0028] The present invention proposes a time series anomaly detection evaluation method and system based on operation and maintenance attention, which realizes the evaluation of the detection effect of the anomaly detection model by simulating the attention changes of the operation and maintenance personnel when monitoring the output of the anomaly detection model and handling abnormal alarms during the actual operation and maintenance process. The method of the present invention can maintain the effectiveness of anomaly detection evaluation under the conditions of long-term anomalies and fragmented detection results, and help operation and maintenance personnel to achieve the performance ranking of anomaly detection models and the selection of the best anomaly detection model in complex scenarios.

[0029] First of all, it is necessary to define the operation and maintenance attention curve in the present invention: the operation and maintenance attention curve simulates the behavior model of the operation and maintenance personnel monitoring the output of the anomaly detection model and handling abnormal alarms during the actual operation and maintenance process. Usually, the operation and maintenance personnel deploy anomaly detection models for key performance indicators (KPIs) in the system, such as business volume and success rate, and check the operating status of the system when the anomaly detection model issues an abnormal alarm to verify whether a fault has occurred. In this process, when the anomaly detection model does not report an abnormal point, the operation and maintenance personnel do not pay attention to the KPI. If the anomaly detection model reports an abnormal point at a certain moment, the operation and maintenance personnel immediately pay close attention to the KPI and begin to check the operating status of the system. This stage is called the "discovery period" of the anomaly. Next, the operation and maintenance personnel begin to operate to handle the anomaly. At the same time, the anomaly detection model continuously or intermittently generates abnormal points. The operation and maintenance personnel maintain a certain degree of attention to the current abnormal event to handle the abnormal event. This stage is called the "persistence period" of the anomaly. Finally, after a certain amount of processing, the abnormal situation is repaired, and the anomaly detection model stops reporting abnormal points. The operation and maintenance personnel will still pay attention to this abnormal event for a period of time to determine that the abnormal situation has indeed been fully restored. This stage is called the "observation period" of the abnormality. At this stage, the operation and maintenance personnel's attention to the current abnormal event will gradually fade, and eventually they will no longer pay attention to the KPI that has returned to normal. For abnormal false alarms, the above operation and maintenance attention stage still applies. The operation and maintenance personnel will be triggered by the generation of false alarms. When the false alarms continue or appear in a concentrated stage, they will check whether there are problems with the system. Finally, after the anomaly detection model stops reporting abnormalities for a period of time, they will gradually reduce their attention to the abnormal event and end their observation of the system status.

[0030] Based on the motivation of modeling the behavior of maintenance personnel when monitoring and handling anomalies, and the fitting of the anomaly discovery period, duration period, and observation period, the present invention defines a set of maintenance attention curve models, as shown below:

[0031] Where j represents the distance between the current position and the abnormal start position, ω(·) represents the operation and maintenance attention function during the discovery period and the duration period, and γ(·) represents the operation and maintenance attention function during the observation period. dis , l dur and l obs Respectively represent the length of the discovery period, the duration period and the observation period. Custom parameter l dis With l obs They are used to control the rate of attention decline during the discovery period and the observation period. dur It is not a custom parameter, but the total time of the abnormal point minus l dis Perform calculations.

[0032] Figure 1A schematic diagram of an abnormal event p1 and its operation and maintenance attention curve is shown. Figure 1 In the above example, p1 is an independent and continuous abnormal event. The operation and maintenance attention function ω(·) during the discovery period and the duration period can use a reversed and scaled sigmoid function. The calculation formula of ω(·) is as follows: The calculation formula of sigmoid(·) is as follows: b dur is the lower bound of the duration of attention, l dis is the length of the discovery period. Figure 1 As shown in Figure 2, when the first anomaly appears, the operation and maintenance attention is set to 1. During the entire discovery period, the operation and maintenance attention quickly drops to close to b dur During the duration, the operation and maintenance attention decreases very slowly and never falls below b dur The calculation formula of the operation and maintenance attention function γ(·) during the observation period is as follows: Among them, l obs It indicates the length of the observation period. The operation and maintenance attention during the observation period starts from the value at the end of the duration period and gradually decreases to 0.

[0033] like Figure 2 As shown, the present invention provides a time series anomaly detection and evaluation method based on operation and maintenance attention, which includes:

[0034] Step 1: Obtain the time series of performance indicators over a period of time X = {X0, x1, ..., x T-1} and the corresponding ground truth labels Y = {y0, y1, ..., y T-1}, T is the total number of time points in the time series X, y t ∈{0,1},t∈{0,1,…,T-1},y t = 0 (or 1) indicates that the state of the time series at time point t is normal (or abnormal), and then the anomaly detection model is used to perform anomaly detection on the time series X and output the anomaly detection result (or 1) indicates that the anomaly detection result of the anomaly detection model at time point t is normal (or abnormal);

[0035] Step 2: Calculate the ground truth labels Y and anomaly detection results one by one The operation attention of each anomaly point in is used to predict the operation attention of all time points within the next observation period, thereby obtaining the ground truth label Y and the anomaly detection result. The operation and maintenance attention curve is used to evaluate the anomaly detection results of the anomaly detection model.

[0036] Using the method of the present invention, on the one hand, the operation and maintenance attention curve can be calculated in the case of fragmented detection results; on the other hand, since the method uses a point-by-point calculation method from front to back to calculate the operation and maintenance attention, there is no need to know the future ground truth labels and anomaly detection model output results. This method can be used for the performance evaluation of anomaly detection models in online applications.

[0037] Since the detection results output by the anomaly detection model are discrete, the detected abnormal events may be fragmented. That is, a ground truth abnormal event may be detected by multiple detection result events. In order to calculate the operation and maintenance attention curve in this case, the present invention proposes an online (front-to-backward) operation and maintenance attention calculation method. In step 2, the ground truth label Y and the anomaly detection result are calculated one by one. The operation attention of each anomaly point in is used to predict the operation attention of all time points within the next observation period, thereby obtaining the ground truth label Y and the anomaly detection result. The operation and maintenance attention curve can further include:

[0038] Step 21, read Y (or ) at the first time point in the

[0039] Step 22, determine whether the read state is abnormal, if yes, then the time point u corresponding to the read state is an abnormal point, and continue to step 23; if no, go to step 26;

[0040] Step 23, determine whether the abnormal point is the first abnormal point of the abnormal event, if yes, proceed to step 24; if no, proceed to step 25;

[0041] Step 24: Set the abnormal start time point p start , abnormal end time point p end are initialized to the time point of the abnormal point, and the operation and maintenance attention of the abnormal point at time point u is set: I(u) = 1, I(u) is the operation and maintenance attention of the abnormal point at time point u, and then calculate the abnormal end time point p end The operation and maintenance attention of all time points within an observation period of the starting point is: I(v) = I(u)*γ(vp end ), where v is the abnormal end time point p end The length of an observation period is l obs A time point within, I(v) is the operation and maintenance attention at time point v, and finally turns to step 26;

[0042] Step 25: Determine the time point u of the abnormal point and the abnormal end time point p endIs the time difference less than the length of an observation period? If so, it means that the abnormal point and the existing abnormal point belong to the same abnormal event. Calculate and save the operation and maintenance attention of the abnormal point at time point u: I(u) = ω(up start ), and then set the abnormal end time point p end Update to the time point u of the abnormal point, calculate and save the abnormal end time point p end The operation and maintenance attention of all time points within an observation period of the starting point is: I(v) = I(u)*γ(vp end ), and finally go to step 26; if not, it means that the abnormal point belongs to a new abnormal event, and does not belong to the same abnormal event as the existing abnormal points. The abnormal point is recorded as the first abnormal point of the abnormal event, and go to step 24;

[0043] Step 26: Determine whether Y has been read (or ) at all time points. If not, continue to read Y (or ), and then go to step 22; if yes, this process ends.

[0044] It should be emphasized that for the operation and maintenance attention at a certain point in time, if the value has been saved, the newly calculated operation and maintenance attention will overwrite the saved operation and maintenance attention.

[0045] Figure 3 A schematic diagram of the operation and maintenance attention curve of a fragmented detection result is shown. Figure 3 It can be seen that in the output of the anomaly detection model, the abnormal interruption point will cause the attention curve to decrease, but it will not drop to 0 immediately. If new abnormal points continue to be generated during the observation period, the fragmented events are merged. On the contrary, if a new abnormal point is re-output after the end of the observation period, its operation and maintenance attention is calculated as a new independent event. Figure 3 In the figure, p1, p2, p3, and p4 belong to the same abnormal event, and p5 belongs to a new independent abnormal event.

[0046] Using ground truth labels Y and anomaly detection results in performance evaluation of time series anomaly detection models For comparison, we can obtain the precision (P), recall (R) and F1 score (F1) of the evaluation results. For the basic binary classification task, the calculation method of the evaluation results is as follows: Among them, TP, FP and FN represent true positives, false positives and false negatives in the basic binary classification task evaluation method. This basic method does not consider the continuity of time series anomalies in the time series, so it is not completely suitable for the performance evaluation of time series anomaly detection models. In the present invention, TP based on the area under the operation and maintenance attention curve can be used.oi , FP oi and FN oi Instead of TP, FP and FN in the basic method, we can obtain the performance evaluation results of the time series anomaly detection model. Figure 4 As shown, in step 2, evaluating the anomaly detection result of the anomaly detection model may further include:

[0047] Step A1: Based on the ground truth label Y and anomaly detection results The overlapping area of ​​the operation and maintenance attention curve and the area under the curve are used to calculate the true positive area TP based on the operation and maintenance attention curve. oi , false positive area FP oi and false negative area FN oi ,in, FN oi =AUC(I)-TP oi , I are the anomaly detection results Operational attention curve for the ground truth label Y, where AUC(·) represents the area under the curve operation, and min(·) represents the new curve obtained by taking the smaller value from the two operational attention curves at each time point;

[0048] Step A2: Calculate the precision P based on operation and maintenance attention oi and recall R oi :

[0049] Step A3: Calculate the F1 score based on operation and maintenance attention oi :

[0050] Figure 5 Shows a TP calculated based on the area under the operation and maintenance attention curve oi , FP oi and FN oi Schematic diagram of , where gt represents the ground truth label and pred represents the anomaly detection result of the anomaly detection model.

[0051] Length of discovery period dis , observation period length l obs and the lower bound of persistence b dur When setting these parameters, one method may be to set them based on experience, for example, dis Set to 5 minutes, obs Another method is to set it according to the historical average abnormal event length of the performance indicator being tested. disSet it to 1 / 4 of the average length of historical abnormal events, and set l obs is set to be equal to the average length of historical abnormal events. In the two setting methods, b dur Both can be set to 0.5.

[0052] like Figure 6 As shown, the present invention provides a time series anomaly detection and evaluation system based on operation and maintenance attention, including:

[0053] A time series acquisition device is used to obtain the time series X = {X0, x1, ..., xT-1} of the performance indicators over a period of time and the corresponding ground truth labels Y = {y0, y1, ..., yT-1}, where T is the total number of time points in the time series X, and y t ∈{0,1},t∈{0,1,…,T-1},y t = 0 (or 1) indicates that the state of the time series at time point t is normal (or abnormal), and then the anomaly detection model is used to perform anomaly detection on the time series X and output the anomaly detection result (or 1) indicates that the anomaly detection result of the anomaly detection model at time point t is normal (or abnormal);

[0054] Anomaly detection evaluation device, which is used to calculate the ground truth label Y and anomaly detection results one by one The operation attention of each anomaly point in is used to predict the operation attention of all time points within the next observation period, thereby obtaining the ground truth label Y and the anomaly detection result. The operation and maintenance attention curve is used to evaluate the anomaly detection results of the anomaly detection model.

[0055] See also Figure 7 , Figure 7 7 is a structural block diagram of a computing device 700 shown in an exemplary embodiment of this specification. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0056] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like.

[0057] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0058] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 700 may also be a mobile or stationary server or a cloud server, etc.

[0059] The processor 720 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned time series anomaly detection and evaluation method based on operation and maintenance attention.

[0060] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the time series anomaly detection and evaluation method based on operation and maintenance attention belong to the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the time series anomaly detection and evaluation method based on operation and maintenance attention.

[0061] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned time series anomaly detection and evaluation method based on operation and maintenance attention.

[0062] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the above-mentioned time series anomaly detection and evaluation method based on operation and maintenance attention belong to the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the above-mentioned time series anomaly detection and evaluation method or system based on operation and maintenance attention.

[0063] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned time series anomaly detection and evaluation method based on operation and maintenance attention.

[0064] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the time series anomaly detection and evaluation method based on operation and maintenance attention belong to the same concept. For details not described in detail in the technical scheme of the computer program, please refer to the description of the technical scheme of the time series anomaly detection and evaluation method or system based on operation and maintenance attention.

[0065] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0066] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0067] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A time series anomaly detection and evaluation method based on operation and maintenance attention, characterized in that: Included are: Step 1: Obtain the time series X = {X0, x1, ..., xT-1} of the performance indicators over a period of time and the corresponding ground truth labels Y = {y0, y1, ..., yT-1}, where T is the total number of time points in the time series X, and y t ∈{0,1},t∈{0,1,…,T-1},y t = 0 or 1 indicates that the state of the time series at time point t is normal or abnormal. Then the anomaly detection model is used to detect anomalies in the time series X and output the anomaly detection results. or 1 indicates that the anomaly detection result of the anomaly detection model at time point t is normal or abnormal; Step 2: Calculate the ground truth labels Y and anomaly detection results one by one The operation attention of each anomaly point in is used to predict the operation attention of all time points within the next observation period, thereby obtaining the ground truth label Y and the anomaly detection result. The operation and maintenance attention curve is used to evaluate the anomaly detection results of the anomaly detection model.

2. The method according to claim 1, characterized in that In step 2, the ground truth labels Y and anomaly detection results are calculated one by one The operation attention of each anomaly point in is used to predict the operation attention of all time points within the next observation period, thereby obtaining the ground truth label Y and the anomaly detection result. The operation and maintenance attention curve further includes: Step 21, read Y or The state at the first time point in ; Step 22, determine whether the read state is abnormal, if yes, then the time point u corresponding to the read state is an abnormal point, and continue to step 23; if no, go to step 26; Step 23, determine whether the abnormal point is the first abnormal point of the abnormal event, if yes, proceed to step 24; if no, proceed to step 25; Step 24: Set the abnormal start time point p start , abnormal end time point p end are initialized to the time point of the abnormal point, and the operation and maintenance attention of the abnormal point at time point u is set: I(u) = 1, I(u) is the operation and maintenance attention of the abnormal point at time point u, and then calculate the abnormal end time point p end The operation and maintenance attention of all time points within an observation period of the starting point is: I(v) = I(u)*γ(vp end ), where v is the abnormal end time point p end The length of an observation period is l obs A time point within, I(v) is the operation and maintenance attention at time point v, γ(·) is the operation and maintenance attention function during the observation period, and finally turn to step 26: Step 25: Determine the time point u of the abnormal point and the abnormal end time point p end Is the time difference less than the length of an observation period? If so, calculate and save the operation and maintenance attention of the abnormal point at time point u: I(u) = ω(u-ps ta rt), ω(·) is the operation and maintenance attention function of the discovery period-continuation period, and then the abnormal end time point p end Update to the time point u of the abnormal point, calculate and save the abnormal end time point p end The operation and maintenance attention of all time points within an observation period of the starting point is: I(v) = I(u)*γ(vp end ), and finally go to step 26; if not, record the abnormal point as the first abnormal point of the abnormal event, and go to step 24; Step 26: Determine whether the reading of Y or If not, continue to read Y or The state at the next time point in the process is determined, and then the process goes to step 22; if yes, the process ends.

3. The method according to claim 2, characterized in that The calculation formula of the operation and maintenance attention function ω(·) during the discovery period to the duration period is as follows: The calculation formula of sigmoid(·) is as follows: b dur is the lower bound of the duration of attention, l dis is the length of the discovery period, The calculation formula of the operation and maintenance attention function γ(·) during the observation period is as follows:

4. The method according to claim 3, characterized in that Length of discovery period dis , observation period length l obs and the lower bound of persistence b dur The parameter settings include: Will l dis Set to 5 minutes, obs Set to 15 minutes, dur Set to 0.5, or, It is set according to the historical average abnormal event length of the detected performance indicator. dis Set it to 1 / 4 of the average length of historical abnormal events, and set l obs Set it to be equal to the average length of historical abnormal events, and set b dur Set to 0.

5.

5. The method according to claim 1, characterized in that: In step 2, the anomaly detection results of the anomaly detection model are evaluated, further including: Step A1: Based on the ground truth label Y and anomaly detection results The overlapping area of ​​the operation and maintenance attention curve and the area under the curve are used to calculate the true positive area TP based on the operation and maintenance attention curve. oi , false positive area FP oi and false negative area FN oi ,in, FN oi =AUC(I)-TP oi , I are the anomaly detection results Operational attention curve for the ground truth label Y, where AUC(·) represents the area under the curve operation, and min(·) represents the new curve obtained by taking the smaller value from the two operational attention curves at each time point; Step A2: Calculate the precision P based on operation and maintenance attention oi and recall R oi : Step A3: Calculate the F1 score based on operation and maintenance attention oi :

6. A time series anomaly detection and evaluation system based on operation and maintenance attention, characterized in that: Included are: A time series acquisition device is used to obtain the time series X = {X0, x1, ..., xT-1} of the performance indicators over a period of time and the corresponding ground truth labels Y = {y0, y1, ..., yT-1}, where T is the total number of time points in the time series X, and y t ∈{0,1},t∈{0,1,…,T-1},y t = 0 or 1 indicates that the state of the time series at time point t is normal or abnormal. Then the anomaly detection model is used to detect anomalies in the time series X and output the anomaly detection results. or 1 indicates that the anomaly detection result of the anomaly detection model at time point t is normal or abnormal; Anomaly detection evaluation device, which is used to calculate the ground truth label Y and anomaly detection results one by one The operation attention of each anomaly point in is used to predict the operation attention of all time points within the next observation period, thereby obtaining the ground truth label Y and the anomaly detection result. The operation and maintenance attention curve is used to evaluate the anomaly detection results of the anomaly detection model.

7. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the time series anomaly detection and evaluation method based on operation and maintenance attention are implemented as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the time series anomaly detection and evaluation method based on operation and maintenance attention as described in any one of claims 1-5.

Images