A dam safety monitoring data anomaly detection method based on LV-DBSCAN

Abstract

The application discloses a kind of dam safety monitoring data anomaly detection method based on LV-DBSCAN.It includes the following steps, step S1: input monitoring data, use LV to extract abnormal points;Step S2: use DBSCAN to extract abnormal points;Step S3: the intersection of abnormal points in step S1 and S2 is defined as accurate abnormal point, the point in the difference set of Step1 and Step2 is defined as local suspicious point, the point in the difference set of step S2 and S1 is defined as global suspicious point;Step S4: the confidence interval in step S1 is enlarged to α+β, whether local suspicious point is abnormal point is judged, if yes, it is abnormal data, otherwise, it is normal point;Step S5: the confidence interval in step S1 is reduced to α-β, whether global suspicious point is abnormal point is judged, if yes, it is abnormal data;Step S6: all abnormal points in step S3-5 are comprehensively processed.The application has the advantages of improving accuracy and reducing misjudgment rate.

Description

Technical Field

[0001] This invention relates to the technical field of anomaly detection in dam safety monitoring data, and more specifically, to a method for anomaly detection in dam safety monitoring data based on LV-DBSCAN. Background Technology

[0002] The acquisition of dam safety monitoring data is inevitably affected by factors such as human error, external interference, and equipment failure, which often results in abnormal measurements in the original monitoring data sequence, greatly affecting the reliability and accuracy of dam safety monitoring data analysis.

[0003] The identification, analysis, and processing of anomalies are fundamental tasks in monitoring data analysis and also one of the main challenges. Commonly used methods for detecting anomalies in dam safety monitoring can be categorized into statistical probability methods, wavelet analysis methods, spatiotemporal discriminant methods, and mathematical modeling methods. Different types of methods have different focuses, leading to limited applicability. Generally, for stationary data sequences, statistical probability methods can effectively identify anomalies that are significantly different from the overall data sequence. However, if the monitored effect quantity shows an increasing trend, the condition of independent and identically distributed data sequences is not met, making statistical probability methods unsuitable. For step anomalies, which may be caused by sudden events such as earthquakes or by missed data due to instrument damage, there are often few precursory indicators in the dam's environmental and effect quantities, making real-time judgment difficult. When using spatiotemporal discriminant methods, the window can easily misjudge the first few values ​​after the step due to the large difference between the points before and after the step.

[0004] Therefore, it is imperative to develop a method for detecting anomalies in dam safety monitoring data that overcomes the difficulty of ensuring complete identification of outliers using a single method, while also achieving high accuracy and low false positive rate. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for detecting anomalies in dam safety monitoring data based on LV-DBSCAN. This method takes into account both the overall and local characteristics of the data, and integrates the Local Variation Anomaly Coefficient (LV) method and the Density Clustering Algorithm (DBSCAN). It has high accuracy and low false positive rate for both stationary and non-stationary data.

[0006] To achieve the above objectives, the technical solution of the present invention is: a method for detecting anomalies in dam safety monitoring data based on LV-DBSCAN, characterized by comprising the following steps,

[0007] Step S1: Input monitoring data and use the Local Variation Coefficient (LV) method to extract outliers;

[0008] Step S2: Input monitoring data and use the density clustering algorithm DBSCAN to extract outliers;

[0009] Step S3: Take the intersection of the anomalies in Step S1 and Step S2 and define them as accurate anomalies. Define the points in the difference set between Step 1 and Step 2 as local suspicious points, and define the points in the difference set between Step S2 and Step S1 as global suspicious points.

[0010] Step S4: For local suspicious points, expand the confidence interval of the local change anomaly coefficient method in step S1 to α+β, and determine whether the suspicious point is still an anomaly. If it is, list it as abnormal data; otherwise, it is a normal point.

[0011] Step S5: For globally suspicious points, narrow the confidence interval in the local change anomaly coefficient method in step S1 to α-β, and determine whether the suspicious point is an anomaly. If it is, list it as abnormal data; otherwise, it is a normal point.

[0012] Step S6: Integrate all anomalies from steps S3 to S5 to achieve anomaly detection.

[0013] In the above technical solution, step S1 uses the Local Variation Coefficient (LV) method to extract outliers, specifically including the following steps:

[0014] Step S11: Assume a set of original monitoring data sequence {x1, x2, ... x t-k ,…,x t-1 The data to be detected is x. t Based on the changing patterns of historical data, the selected threshold is {b}. U b L};

[0015] Step S12: Calculate the value to be detected, x t rate of change at x′ t =(x t -x t-i ) / Δt, if x t-i If it is determined to be an outlier, then take x. t-i-1 Calculate the rate of change x′ t The process continues until i+1>m, where m is the window width. If there is no data or all data in the window is abnormal, the value to be detected is directly determined as x. t This is normal;

[0016] Step S13: Calculate the rate of change and the corresponding upper and lower thresholds b normalization coefficient P according to formula (1). If P is between 0 and 1, the measured value is judged to be a normal value, and the abnormality coefficient of the normal value is set to 0. If P is greater than 1 or less than 0, the measured value is judged to be an abnormal value, and P is used as x. t The anomaly coefficient;

[0017]

[0018] Step S14: Take the data at time t+1 as the data to be detected, and repeat steps S11 to S13. That is, whenever real-time data (the value to be detected) arrives, steps S11 to S13 can be repeated to achieve anomaly detection of real-time data.

[0019] In the above technical solution, the threshold b calculation step in step S11 is as follows:

[0020] Step S111: Assume a set of original monitoring data sequence {x1, x2, ..., x...} t}, according to x′ t =(x t -x t-i The rate of change sequence {x′2, x′3, ..., x′} is obtained by calculating Δy / Δy. t};

[0021] Step S112: Perform frequency analysis on the rate of change sequence to obtain the sequence probability density function f(x′) and cumulative probability distribution function F(x′); the frequency analysis method uses the kernel density estimation method;

[0022] Step S113: Based on the principle of low probability, set the confidence level to α, and... and Distribution as upper and lower bounds of threshold b {b U b L}

[0023] In the above technical solution, in step S12, the window length m is recommended to be 30 days; it can be appropriately shortened during the construction period when the observation frequency is high, and can be appropriately increased during the operation period when the measured values ​​are relatively stable.

[0024] In the above technical solution, step S2, which uses a density clustering algorithm to extract outliers, specifically includes the following steps:

[0025] Step S21: Start with an unvisited arbitrary data point. The neighborhood of this point is defined by the distance epsilon (i.e., all points within the ε-distance range of this point are neighborhood points). If there are a sufficient number of points in this neighborhood (i.e., minPoints), the clustering process begins, and the current data point becomes the first point in the new cluster; otherwise, the point will be marked as noise. In both cases, the point will be marked as "visited".

[0026] Step S22: For the first point in the new cluster, the points in its ε-distance neighborhood will also become part of the same cluster; this process makes all points in the ε-neighborhood belong to the same cluster, and then repeat the above process for all new points added to the cluster;

[0027] Step S23: repeat step S21-step S22 until all points are marked as visited stop;

[0028] Step S24: the cluster result exists data amount ratio less than 5% cluster and noise point is judged as abnormal data, and the rest is normal data.

[0029] Compared with the prior art, the present application has the following advantages:

[0030] (1) The present application considers the overall and local characteristics of data as a whole, and combines the advantages of LV method and DBSCAN method, and has good abnormal detection effect;

[0031] (2) The local method and the overall method of the present application are mainly dependent on the local variation anomaly coefficient method when they are cooperatively distinguished, and the change in the strictness of data anomaly discrimination is realized by modifying the confidence degree, and the statistical probability method or other types of clustering algorithm can also be used in the selection of the overall method. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A flow chart of a dam safety monitoring data anomaly detection method based on LV-DBSCAN is provided in the present application.

[0033] Figure 2 A comparison chart of different anomaly detection methods for downstream displacement monitoring data of a certain concrete gravity dam deformation measuring point in the embodiment of the present application. DETAILED DESCRIPTION

[0034] The implementation of the present application will be described in detail below with reference to the accompanying drawings, but they do not constitute a limitation on the present application, but only serve as an example. At the same time, the advantages of the present application are made more clear and easy to understand through the description.

[0035] As shown in the drawings, the present application is a kind of based on LV-DBSCAN's dam safety monitoring data anomaly detection method, including the following steps:

[0036] Step S1: input monitoring data, and use local variation coefficient method (LV) to extract abnormal points;

[0037] Step S2: input monitoring data, and use DBSCAN method to extract abnormal points;

[0038] Step S3: take the intersection of the abnormal points in step S1 and step S2, define as accurate abnormal points, the difference set of Step1 and Step2 is defined as local suspicious points, and the difference set of step S2 and step S1 is defined as global suspicious points;

[0039] Step S4: For the local suspicious point, the confidence interval in the local variation anomaly coefficient method in step S1 is expanded to a+β, to determine whether the suspicious point is still an abnormal point, if yes, list it as an abnormal data, otherwise, it is a normal point;

[0040] Step S5: For the global suspicious point, the confidence interval in the local variation anomaly coefficient method in step S1 is reduced to a-β, to determine whether the suspicious point is an abnormal point, if yes, list it as an abnormal data, otherwise, it is a normal point;

[0041] Step S6: Integrate all abnormal points in steps S3-S6, thus realizing the abnormality detection (as shown in FIG. 1). Figure 1

[0042] Wherein, the extraction of abnormal points using the local variation coefficient method (LV) comprises the following steps:

[0043] Step S11: Assume a group of original monitoring data sequences {x1, x2, … x t-k , …, x t-1}, the data to be detected is x t . According to the change rule of historical data, the threshold values selected are {b U , b L};

[0044] Step S12: Calculate the change rate x′ t = (x t -x t ) / Δt at the detected value x t-i , if x t-i-1 is determined as an abnormal value, calculate the change rate x′ t , until i+1>m, m is the window width, if there is no data or all are abnormal data in the window, directly determine that the detected value x t is normal;

[0045] Step S13: Calculate the change rate and the corresponding upper and lower threshold b t normalization coefficient P, if P is between 0 and 1, determine that the detected value is a normal value, set the abnormal coefficient of the normal value as 0, if P is greater than 1 or less than 0, determine that the detected value is an abnormal value, take P as the abnormal coefficient of x t ;

[0046]

[0047] Step S14: Take the data at t+1 time as the detected data, repeat steps S11-S13, that is, whenever real-time data (detected value) comes, repeat steps S11-S13 to realize the abnormality detection of real-time data.

[0048] ​Wherein, the threshold b calculation step in step S11 is specifically as follows:

[0049] Step S111: assuming a set of original monitoring data sequences {x1, x2, …, x t}, the rate of change sequence {z'2, x'3, …, x'N} is calculated according to the formula x' t = {x t -x t-i ) / Δt; t

[0050] Step S112: frequency analysis is performed on the rate of change sequence to obtain the sequence probability density function f(x') and the cumulative probability distribution function F(x'); the probability distribution fitting method uses the kernel density estimation method;

[0051] Step S113: according to the small probability principle, the confidence level is set to α, and the upper and lower limits {b U , b L} of the threshold b are set as the and distributions.

[0052] Wherein, the window length m in step S12 is recommended to be 30 days; the observation frequency during the construction period is high, and can be appropriately shortened; the measurement value during the operation period is relatively stable, and can be appropriately increased.

[0053] Wherein, in step S2, the DBSCAN method is used to extract the abnormal points (wherein the density clustering algorithm DBSCAN is prior art), and specifically includes the following steps:

[0054] Step S21: starting from an arbitrary data point that has not been visited, the neighborhood of this point is defined by the distance epsilon (that is, all points within the ε distance range of the point are neighborhood points), if there are a sufficient number of points (that is, minPoints) in the neighborhood (this is a parameter of the density clustering algorithm, and is defined according to the data characteristics during use), the clustering process starts, and the current data point becomes the first point in the new cluster. Otherwise, the point will be marked as noise. In both cases, the point will be marked as “visited”;

[0055] Step S22: for the first point in the new cluster, the points in the ε distance neighborhood of the point will also become part of the same cluster. This process makes all points in the ε neighborhood belong to the same cluster, and then repeats the above process for all new points added to the cluster;

[0056] Step S23: repeat steps S21-S22 until all points are marked as visited to stop;

[0057] Step S24: the clusters with less data and the noise points in the clustering results are determined as abnormal data, and the rest are normal data.​

[0058] Embodiment: Taking the downstream displacement monitoring data of a certain concrete gravity dam deformation measuring point as an example, according to the autocorrelation of the data sequence, the window size of the multiple local anomaly coefficient method is 5, the confidence of the local variation anomaly coefficient method is a = 0.90, the neighborhood distance of the DBSCAN algorithm is 0.1, the minimum number of neighborhood points is 6, and the additional parameter β of the LV-DBSCAN method is 0.05. Figure 2 The abnormal value detection results of various methods are shown in Table 1.

[0059] Table 1 Comparison of different abnormal detection methods for the downstream displacement monitoring data of a certain concrete gravity dam deformation measuring point 1

[0060] Method Total number of abnormal data Identifying abnormal data Misjudged as abnormal Misjudged as normal Accuracy (%) Recall (%) F1 score Local variation anomaly coefficient method 20 20 13 0 58.8 100.0 0.74 DBSCAN 20 14 3 6 82.4 70.0 0.76 The LV-DBSCAN of the present application 20 18 4 2 81.8 90.0 0.86

[0061] Note: The accuracy in the table is the proportion of data that is actually abnormal in the total number of identified abnormalities (for example, the accuracy of the DBSCAN method is 14 / (14+3) x 100%); The recall rate is the proportion of data that is actually abnormal in the total data (for example, the recall rate of the DBSCAN method is 14 / 20 x 100%; The F1 score is 2 x 81.8% x 90.0% / (81.8% + 90.0%)).

[0062] From the above Table 1, it can be seen that the accuracy of the present application is higher, much higher than the local variation anomaly coefficient method, and the recall rate is significantly higher than the DBSCAN method. In summary, the F1 score of the present application is higher, and the comprehensive index is better than the existing local variation anomaly coefficient method and DBSCAN method.

[0063] From the above Table 1 and Figure 2 It can be seen that: in this embodiment, the downstream displacement monitoring data of a certain concrete gravity dam deformation measuring point uses different abnormal detection methods (local variation anomaly coefficient method, DBSCAN, LV-DBSCAN of the present application), and the identified abnormal data is partially the same and partially different. Among them, the abnormal data points identified by DBSCAN and LV-DBSCAN of the present application are mostly the same, the number of abnormal data points identified is large, and the accuracy is high, but the number of abnormal data points identified by LV-DBSCAN of the present application is more than that of DBSCAN, and the misjudgment of DBSCAN as normal is more; The local variation anomaly coefficient method identifies a large number of abnormal data points, but it misjudges a large number of normal data points as abnormal, and the accuracy is low. Therefore, compared with the local variation anomaly coefficient method and DBSCAN, the method of the present application has good abnormal detection effect.

[0064] The above merely illustrates the specific embodiments of the present application, and it should be noted that any changes or replacements within the technical scope disclosed by the present application can be easily conceived by any person skilled in the art, and should be covered within the protection scope of the present application, and the rest not described in detail belong to the prior art.

Claims

1. A method for detecting anomalies in dam safety monitoring data based on LV-DBSCAN, characterized in that: Includes the following steps, Step S1: Input monitoring data and use the Local Variation Coefficient (LV) method to extract outliers; Step S2: Input monitoring data and use the density clustering algorithm DBSCAN to extract outliers; Step S3: Take the intersection of the anomalies in Step S1 and Step S2 and define them as accurate anomalies. Define the points in the difference set between Step 1 and Step 2 as local suspicious points, and define the points in the difference set between Step S2 and Step S1 as global suspicious points. Step S4: For local suspicious points, expand the confidence interval of the local change anomaly coefficient method in step S1 to α+β, and determine whether the suspicious point is still an anomaly. If it is, list it as abnormal data; otherwise, it is a normal point. Here, α is the confidence level and β is an additional parameter. Step S5: For globally suspicious points, narrow the confidence interval in the local change anomaly coefficient method in step S1 to α-β, and determine whether the suspicious point is an anomaly. If it is, list it as abnormal data; otherwise, it is a normal point. Step S6: Integrate all anomalies from steps S3 to S5 to achieve anomaly detection; In step S1, outliers are extracted using the Local Variation Coefficient (LV) method, which specifically includes the following steps: Step S11: Assume a set of original monitoring data sequence {x1, x2, ... x t-k ,…,x t-1 The data to be detected is x. t Based on the changing patterns of historical data, the selected threshold is {b}. U b L }; Step S12: Calculate the value to be detected, x t rate of change at x′ t =(x t -x t-i ) / Δt, if x t-i If it is determined to be an outlier, then take x. t-i-1 Calculate the rate of change x′ t The process continues until i+1>m, where m is the window width. If there is no data or all data in the window is abnormal, the value to be detected is directly determined as x. t This is normal; Step S13: Calculate the rate of change and the corresponding upper and lower thresholds b, and the normalization coefficient P. If P is between 0 and 1, the measured value is determined to be a normal value, and the abnormality coefficient of the normal value is set to 0. If P is greater than 1 or less than 0, the measured value is determined to be an abnormal value, and P is used as x. t The anomaly coefficient; Step S14: Take the data at time t+1 as the data to be detected, and repeat steps S11 to S13 to achieve anomaly detection in real-time data.

2. The method for detecting anomalies in dam safety monitoring data based on LV-DBSCAN according to claim 1, characterized in that: In step S11, the threshold b is calculated as follows: Step S111: Assume a set of original monitoring data sequence {x1, x2, ..., x...} t }, according to x′ t =(x t -x t-i The rate of change sequence {x′2, x′3, ..., x′} is obtained by calculating Δt / t. t }; Step S112: Perform frequency analysis on the rate of change sequence to obtain the sequence probability density function f(x′) and cumulative probability distribution function F(x′); the kernel density estimation method is used for probability distribution fitting. Step S113: Based on the principle of small probability, set the confidence level as α, and... and Distribution as upper and lower bounds of threshold b {b U b L } 3. The method for detecting anomalies in dam safety monitoring data based on LV-DBSCAN according to claim 2, characterized in that: In step S12, the window length m is set to 30 days.

4. The method for detecting anomalies in dam safety monitoring data based on LV-DBSCAN according to claim 3, characterized in that: In step S2, outliers are extracted using a density clustering algorithm, which specifically includes the following steps: Step S21: Start with an unvisited arbitrary data point whose neighborhood is defined by distance epsilon. If there are enough points in the neighborhood, the clustering process begins, and the current data point becomes the first point in the new cluster; otherwise, the point will be marked as noise. Step S22: For the first point in the new cluster, the points in its ε-distance neighborhood will also become part of the same cluster; this process makes all points in the ε-neighborhood belong to the same cluster, and then repeat the above process for all new points added to the cluster; Step S23: Repeat steps S21-S22 until all points are marked as visited and stop; Step S24: Clusters with fewer data points and noisy points in the clustering results are identified as abnormal data, while the rest are normal data.

Images