Abnormal data detection method and device

Abstract

The invention relates to an abnormal data detection method and device. The detection method comprises the following steps that S0, a sliding window with a set window length is used for sliding along to-be-detected data for one time according to a set step length, and a sequence formed by data elements in the window obtained after sliding each time serves as a sub-sequence; the method comprises the following steps: S1, for any sequence element in a current sub-sequence, respectively calculating distances between the sequence element and all other sequence elements in the sub-sequence, and counting the number of the distances greater than a safety threshold as a safety degree element of the sequence element; s2, acquiring a set of safety degree elements of all sequence elements in the current sub-sequence as the safety degree of the current sub-sequence; and S3, performing analysis according to the element distribution condition in the safety degree of the current sub-sequence to obtain an outlier abnormal point. The abnormal data in the complex change data can be reliably and effectively detected.

Description

Technical Field

[0001] The present invention relates to the technical field of data mining, and in particular to an abnormal data detection method and device. Background Art

[0002] Data anomaly types can be divided into point anomalies and sequence anomalies. Point anomalies refer to a data point that is significantly different from the data points around it; sequence anomalies refer to a data segment that is significantly different from the data around it. At present, the common methods for data outlier detection mainly include the following.

[0003] 1) Statistical methods. This method is mainly used to detect numerical data. Common methods include the Raida criterion [3-4], Z-Score [5], box plot [6], Grubbs hypothesis test [7], etc. These methods can be used to process disordered or stable data sets, but are not applicable to data sets with large fluctuations.

[0004] 2) Distance-based methods. This method includes the K nearest neighbor classification algorithm [8] and the K means clustering algorithm [9]. The K nearest neighbor classification algorithm is a classification algorithm in supervised learning, and the K means clustering algorithm is a classification algorithm in unsupervised learning. Both methods are based on Euclidean distance for classification. Both methods require the K value to be specified in advance and are sensitive to the shape and size of the cluster, which has limitations.

[0005] 3) Density-based methods. These methods include the Local Outlier Factor

[10] (LOF) algorithm, the DBSCAN

[11] (Density-Based Spatial Clustering of Applications with Noise) algorithm, the COF outlier metric

[12] (Connectivity-Based Outlier Factor, COF) algorithm, and the Symbiotic Organisms Search

[13] (SOS) algorithm. These algorithms do not require the number of clusters to be specified in advance and have good robustness to noise points. However, for data sets with uneven density, it is difficult to select appropriate parameters for the radius and number of neighbor points.

[0006] In summary, statistical methods are not suitable for processing data sets with large fluctuations; distance-based methods require pre-specified K values ​​and are easily affected by the shape and size of clusters, which has certain limitations; density-based methods are more difficult to select appropriate parameters such as radius and number of neighbor points for data sets with uneven density. Statistical, distance-based, and density-based detection algorithms all have their own advantages and disadvantages. For data with complex changes, a single outlier detection algorithm has great limitations and cannot meet the actual needs of data outlier detection. Therefore, a more reliable and effective detection method is needed. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a method and device for detecting abnormal data, which can reliably and effectively detect abnormal data in complex changing data.

[0008] The technical solution adopted by the present invention to solve the technical problem is: to provide an abnormal data detection method, comprising the following steps:

[0009] S0 uses a sliding window with a set window length to slide along the data to be tested once with a set step length, and takes the sequence of data elements in the window obtained after each sliding as a subsequence;

[0010] S1, for any sequence element in the current subsequence, respectively calculates the distance between it and all other sequence elements in the subsequence, and counts the number of distances greater than the safety threshold as the safety element of the sequence element;

[0011] S2 obtains the set of safety elements of all sequence elements in the current subsequence as the safety of the current subsequence;

[0012] S3: Analyze the distribution of elements in the safety level of the current subsequence to obtain outlier points.

[0013] Furthermore, before step S3, the method further includes:

[0014] It is determined whether all the safety elements of the current subsequence are less than or equal to the safety parameter. If so, the process returns to step S0; otherwise, the process goes to step S3.

[0015] Furthermore, step S3 is achieved by performing cluster analysis based on a mean shift clustering algorithm on the safety level of the current subsequence.

[0016] Furthermore, the performing cluster analysis on the safety level of the current subsequence based on a mean shift clustering algorithm includes:

[0017] estimating an optimal bandwidth of a kernel function according to the safety degree of the current subsequence;

[0018] Clustering the safety degree of the current subsequence using a mean shift clustering algorithm according to the optimal bandwidth;

[0019] The cluster where the minimum safety element is located is taken as a normal cluster, and the sequence elements corresponding to the safety elements contained in other clusters are taken as the outliers.

[0020] Furthermore, the kernel function is a Gaussian kernel, and its optimal bandwidth is expressed as

[0021]

[0022] Where h is the optimal bandwidth, is the standard deviation of the safety level, and n is the set window length.

[0023] Furthermore, the method also includes a step of using auxiliary information to determine whether the extracted outlier points are abnormal data.

[0024] Furthermore, the set step length is smaller than the set window length.

[0025] The present invention also provides an abnormal data detection device, comprising:

[0026] A segmentation module, used for sliding a sliding window with a set window length along the data to be tested once with a set step length, and taking a sequence consisting of data elements in the window obtained after each sliding as a subsequence;

[0027] A first calculation module is used to calculate, for any sequence element in the current subsequence, the distance between it and all other sequence elements in the subsequence, and count the number of the distances greater than the safety threshold as the safety element of the sequence element;

[0028] A second calculation module is used to obtain a set of the safety elements of all sequence elements in the current subsequence as the safety of the current subsequence;

[0029] The cluster analysis module is used to analyze and obtain outlier points according to the distribution of elements in the safety degree of the current subsequence.

[0030] Furthermore, it also includes a judgment module for using auxiliary information to judge whether the extracted outlier points are abnormal data.

[0031] Beneficial Effects

[0032] Due to the adoption of the above technical solution, compared with the prior art, the present invention has the following advantages and positive effects: The present invention continuously detects and divides the data to be measured by using a sliding window, calculates a safety value to associate the currently obtained subsequence with a safety threshold, and then combines kernel density estimation and mean shift clustering algorithm to detect outliers for the safety level, solving the problem that the existing abnormal data detection methods are not applicable to processing data with large fluctuations and uneven densities; by independently calculating the parameters required for clustering analysis for different subsequences, fusing kernel density estimation with an adaptive bandwidth and mean shift clustering algorithm, the accuracy of the detection result is increased; by combining auxiliary information to determine whether an outlier is abnormal data, the reliability and accuracy of the detection are further increased. Description of the Drawings

[0033] Figure 1 is a flowchart of the first embodiment of the present invention;

[0034] Figure 2 is a flowchart of outlier detection in the first embodiment of the present invention;

[0035] Figure 3 is a flowchart of outlier marking in the first embodiment of the present invention;

[0036] Figure 4 is a schematic diagram of water level data in the first embodiment of the present invention;

[0037] Figure 5 is a schematic diagram of water level data with abnormal data marked in the first embodiment of the present invention. Specific Embodiments

[0038] The following further elaborates the present invention in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

[0039] The first embodiment of the present invention relates to an abnormal data detection method, as Figure 1 shown, including the following steps:

[0040] Use a sliding window with a length of n and a step size of m (0 < m < n) to perform continuous sliding window detection on the data;

[0041] Form a subsequence from the n data within the sliding window, calculate new associated data according to the subsequence and the safety threshold, and call this associated data the safety level;

[0042] Calculate the optimal bandwidth of the Gaussian kernel function according to the safety level and kernel density estimation;

[0043] Under the condition of the optimal bandwidth of the kernel function, the mean shift clustering algorithm is used to detect outliers in terms of safety, and the outlier set is output;

[0044] Combined with auxiliary information, it is determined whether the outliers are abnormal points, and finally the abnormal data is output.

[0045] First, the relevant definitions are given below, and then this solution is described in detail.

[0046] 1.1 Relevant Definitions

[0047] 1) Data Element

[0048] For a data set with a length of L, any data in it is called a data element, denoted as x i , where the value range of i is 1 ≤ i ≤ L.

[0049] 2) Subsequence

[0050] For a sliding window with a window length of n and a step size of m (0 < m < n), the sequence composed of the data elements within the sliding window is called a subsequence, denoted as S = {s1, s2, …, s n}, s j is a subsequence element, and the value range of j is 1 ≤ j ≤ n. Each time the sliding window slides backward, the subsequence S is updated. When the left edge of the window slides to the data element x i , there is s j = x i+j-1 , then the subsequence at this time is S = {x i , x i+1 , …, x i+n-1}. After the subsequence S is detected, the sliding window slides backward with a step size of m, and the subsequence is updated to S = {x i+m , x i+1+m , …, x i+n-1+m}.

[0051] 3) Difference Sequence

[0052] The set composed of the distances between any subsequence element s j in the subsequence S and all subsequence elements is called the difference sequence of the subsequence element s j , denoted as d j = {|s1 - s j |, |s2 - s j |, …, |s n - s j |}. For any subsequence element s j in the subsequence S, there is exactly one difference sequence d j corresponding to it.

[0053] 4) Set of differences

[0054] The set of difference sequences corresponding to all subsequence elements in subsequence S is called the difference set of subsequence S, denoted as

[0055] D = {d1, d2, …, d n}.

[0056] 5) Safety threshold

[0057] The parameter used to calculate the safety degree, denoted as T.

[0058] 6) Safety degree

[0059] For the difference sequence d j , the number of elements inside it that are greater than the safety threshold T is denoted as the safety degree element c j . The set

[0060] {c1, c2, …, c n} is called the safety degree of subsequence S, denoted as C, that is, C = {c1, c2, …, c n}.

[0061] 7) Safety degree parameter

[0062] The parameter used to determine whether clustering of the safety degree is required, denoted as N.

[0063] 8) Auxiliary information

[0064] When judging whether an outlier is an anomaly, other data information is usually referred to. Such information is called auxiliary information. If the auxiliary information can prove that the outlier is not an anomaly, it is said that the outlier is supported by the auxiliary information.

[0065] 1.2 Specific solution

[0066] The technical solution is generally divided into two parts: outlier detection and anomaly marking, and its flowchart is as shown in Figure 2 and Figure 3 .

[0067] 1) Outlier detection

[0068] Set the safety threshold T, the safety degree parameter N, and use a sliding window with a window length of n and a step size of m (0 < m < n) to slide along the data to detect outliers. Since the numerical distribution of the data elements within the window is different after each slide, the outlier detection of different subsequences is independent, and the parameters required for clustering need to be calculated independently. For the detected outliers, store them in the outlier set. After all the data is detected, input the outlier set into the next program for detailed verification.

[0069] (a) Generate subsequence

[0070] The n data elements in the sliding window constitute a subsequence S = {s1, s2, ..., s n}.

[0071] (b) Calculation of safety

[0072] Calculate the difference set D of the subsequence S = {d1, d2, ..., d n}, and calculate the safety degree C of the subsequence S according to the safety threshold T = {c1,c2,…,c n}. If max{c1,c2,…,c n}>N, then go to step (c); if max{c1,c2,…,c n}

[0073] ≤N, the window slides backwards and re-enters step (a).

[0074] (c) Calculate the optimal bandwidth of the kernel function

[0075] The Gaussian kernel function is introduced to improve the mean shift clustering algorithm, and the optimal bandwidth h of the kernel function is calculated according to the safety degree C and the kernel density estimation method. The expression of h is:

[0076]

[0077] in is the standard deviation of the safety factor C.

[0078] (d) Mean shift clustering

[0079] Under the condition of optimal bandwidth h, the mean shift clustering algorithm is used to cluster the safety degree C. The minimum element of safety degree C min{c1,c2,…,c n The cluster where} is located is defined as the normal cluster, and the corresponding data of other clusters are classified into the outlier set.

[0080] 2) Outlier Marking

[0081] Auxiliary information is introduced to compare the outliers with the auxiliary information at the same time to determine whether the outliers are abnormal data.

[0082] The present embodiment is further described below in conjunction with specific data for outlier detection of water level data. The auxiliary information is gate opening data, and both the water level data and the gate opening data are measured data of a hydrological station. The accuracy of the water level data is centimeters, and the time intervals between adjacent water level data and gate opening data are 1 second. The following is an example of the processing flow of the present invention for outlier detection of water level data.

[0083] The selected time span is from 2023-01-2017:45 to 2023-01-2017:55, with a total of 600 water level data. The water level data are shown in Table 1.

[0084] Table 1 Water level data

[0085]

[0086] The scatter plot of water level data is as follows Figure 4 The gate opening data for the same time period are shown in Table 2.

[0087] Table 2 Gate opening data

[0088]

[0089] Set the sliding window length n = 30, the sliding step length m = 1, the safety threshold T = 0.03, and the safety parameter N = 3, and perform sliding window detection on these 600 water level data. When the window slides to 2023-01-2017:49:59, the expression of the subsequence S is as follows:

[0090] S={0.04,0.04,0.04,…,0.04,0.04,-1.74} (2)

[0091] The difference sequence is calculated as shown in Table 3.

[0092] Table 3 Difference sequence of subsequence S

[0093]

[0094]

[0095] According to the difference sequence in Table 3 and the safety threshold T, the safety degree C can be calculated. The expression is as follows:

[0096] C={1,1,1,…,1,1,29} (3)

[0097] From the above formula, we can see that Max(C)=29, which is greater than the safety parameter N. Therefore, the safety parameter C enters the cluster detection to separate the outliers.

[0098] The optimal bandwidth h = 2.74 is calculated by formula (1) and formula (3). Under the condition of h = 2.74, the clustering result of the mean shift clustering algorithm for the safety degree C is {0,0,0,…,0,0,1}. From the clustering result, we can know that s in the subsequence S 30 =-1.74 is an outlier, and the point is included in the outlier set. The outlier detection of subsequence S is completed, and the window slides backward to continue detection.

[0099] Repeat the above operation until all water level data are detected, and then input the outlier set into the next program to complete the abnormal point marking. Here, the outlier above is taken as an example to describe in detail the process of determining whether an outlier is abnormal.

[0100] The gate opening data at the time corresponding to the outlier point above is queried, and the results are shown in Table 4.

[0101] Table 4 Outlier information

[0102]

[0103] It can be seen from the gate opening that the gate is in a closed state at this time, and the influence of the gate opening and closing on the water level can be excluded. Therefore, the outlier point is not supported by the auxiliary information and is marked as an abnormal point. All the abnormal points detected in the final water level data are shown in Table 5. Figure 5 shown.

[0104] Table 5 Abnormal water level data

[0105]

[0106]

[0107] contrast Figure 4 , Figure 5 It can be seen that after the water level data is processed by the outlier detection algorithm, all the outliers contained in it are detected; at the same time, the detection algorithm does not make false alarms for normal fluctuations in the water level data. This shows the effectiveness and stability of the method proposed in the present invention for data outlier detection.

[0108] The second embodiment of the present invention relates to an abnormal data detection device, which is used to implement the detection method in the first embodiment, and specifically includes:

[0109] A segmentation module, used for sliding a sliding window with a set window length along the data to be tested once with a set step length, and taking a sequence consisting of data elements in the window obtained after each sliding as a subsequence;

[0110] A first calculation module is used to calculate the distance between any sequence element in the current subsequence and all other sequence elements in the subsequence, and count the number of distances greater than the safety threshold as the safety element of the sequence element;

[0111] The second calculation module is used to obtain a set of safety elements of all sequence elements in the current subsequence as the safety of the current subsequence;

[0112] The cluster analysis module is used to analyze the distribution of elements in the safety degree of the current subsequence and obtain outlier points.

[0113] A judgment module, configured to use auxiliary information to judge whether the extracted outlier is abnormal data.

[0114] When performing abnormal data detection, it can be divided into two stages: outlier detection and outlier marking.

[0115] 1) Outlier detection

[0116] Set a safety threshold T, a safety parameter N, and use a sliding window with a window length of n and a step size of m (0 < m < n) to slide along the data to detect outliers. Since the numerical distribution of the data elements within the window is different after each slide, the outlier detection of different subsequences is independent of each other, and the parameters required for clustering need to be calculated independently. For the detected outliers, store them in the outlier set. After all the data is detected, input the outlier set into the next program for detailed verification.

[0117] (a) Generate subsequences

[0118] The n data elements within the sliding window form a subsequence S = {s1, s2, …, s n}

[0119] (b) Calculate the safety degree

[0120] Calculate the difference set D = {d1, d2, …, d n} of the subsequence S, and calculate the safety degree C = {c1, c2, …, c n} of the subsequence S according to the safety threshold T. If max{c1, c2, …, c n} > N, then enter step (c); if max{c1, c2, …, c n}

[0121] ≤ N, then slide the window backward and re-enter step (a).

[0122] (c) Calculate the optimal bandwidth of the kernel function

[0123] Introduce the Gaussian kernel function to improve the mean shift clustering algorithm, and calculate the optimal bandwidth h of the kernel function according to the safety degree C and the kernel density estimation method. The expression of h is:

[0124]

[0125] where is the standard deviation of the safety degree C.

[0126] (d) Mean shift clustering

[0127] Under the condition of optimal bandwidth h, the mean shift clustering algorithm is used to cluster the safety degree C. The minimum element of safety degree C min{c1,c2,…,c n The cluster where} is located is defined as the normal cluster, and the corresponding data of other clusters are classified into the outlier set.

[0128] 2) Outlier Marking

[0129] Auxiliary information is introduced to compare the outliers with the auxiliary information at the same time to determine whether the outliers are abnormal data.

Claims

1. A method for detecting abnormal data, characterized in that: The following steps are involved: S0 uses a sliding window with a set window length to slide along the data to be tested once with a set step length, and takes the sequence of data elements in the window obtained after each sliding as a subsequence; S1, for any sequence element in the current subsequence, respectively calculates the distance between it and all other sequence elements in the subsequence, and counts the number of distances greater than the safety threshold as the safety element of the sequence element; S2 obtains the set of safety elements of all sequence elements in the current subsequence as the safety of the current subsequence; S3: Analyze the distribution of elements in the safety level of the current subsequence to obtain outlier points.

2. The method according to claim 1, characterized in that Before step S3, the method further includes: It is determined whether all the safety elements of the current subsequence are less than or equal to the safety parameter. If so, the process returns to step S0; otherwise, the process goes to step S3.

3. The method according to claim 1, characterized in that Step S3 is achieved by performing cluster analysis based on a mean shift clustering algorithm on the safety level of the current subsequence.

4. The method according to claim 3, characterized in that The performing cluster analysis on the safety degree of the current subsequence based on a mean shift clustering algorithm comprises: estimating an optimal bandwidth of a kernel function according to the safety degree of the current subsequence; Clustering the safety degree of the current subsequence using a mean shift clustering algorithm according to the optimal bandwidth; The cluster where the minimum safety element is located is taken as a normal cluster, and the sequence elements corresponding to the safety elements contained in other clusters are taken as the outliers.

5. The method according to claim 4, characterized in that The kernel function is a Gaussian kernel, and its optimal bandwidth is expressed as Where h is the optimal bandwidth, is the standard deviation of the safety level, and n is the set window length.

6. The method according to claim 1, characterized in that The method also includes the step of using auxiliary information to determine whether the extracted outlier points are abnormal data.

7. The method according to claim 1, characterized in that The set step length is smaller than the set window length.

8. An abnormal data detection device, characterized in that: include: A segmentation module, used for sliding a sliding window with a set window length along the data to be tested once with a set step length, and taking a sequence consisting of data elements in the window obtained after each sliding as a subsequence; A first calculation module is used to calculate, for any sequence element in the current subsequence, the distance between it and all other sequence elements in the subsequence, and count the number of the distances greater than the safety threshold as the safety element of the sequence element; A second calculation module is used to obtain a set of the safety elements of all sequence elements in the current subsequence as the safety of the current subsequence; The cluster analysis module is used to analyze and obtain outlier points according to the distribution of elements in the safety degree of the current subsequence.

9. The device according to claim 8, characterized in that It also includes a judgment module for using auxiliary information to judge whether the extracted outlier points are abnormal data.

Images