Abnormal data identification method based on K-means algorithm

Abstract

The invention relates to an abnormal data identification method based on a K-means algorithm, and the method comprises the steps: recognizing numerous small clusters through a K-means clustering algorithm, carrying out the statistics of the distribution probability of the number of data objects in each cluster, generating a probability distribution diagram, and taking the probability distribution diagram as a decision diagram. And clearly observing which clusters have the number of data objects obviously smaller than that of other clusters from the decision diagram, so that the clusters are identified as abnormal clusters, and the data objects are identified as abnormal clusters. According to the abnormal data object recognition method, external parameters do not need to be input, and the abnormal recognition result is obtained through the visual information provided by the decision diagram.

Description

Technical Field

[0001] The invention belongs to the field of computer data processing, and in particular relates to an abnormal data recognition method based on a K-means algorithm. Background Art

[0002] Anomaly recognition is of great significance in today's society, promoting progress and development in various fields in improving safety, optimizing efficiency, predicting the future, improving data quality, and supporting decision-making. With the continuous development of technology and the expansion of its application scope, anomaly recognition will continue to play an important role and bring more benefits to human society.

[0003] At present, thousands of anomaly recognition algorithms have been proposed. They can be roughly divided into statistics-based, distance-based, deep learning-based, and ensemble-based algorithms. Specifically, statistics-based algorithms use the statistical characteristics of data to identify outliers. Common statistical information includes mean, standard deviation, median, percentile, etc. Distance-based algorithms identify outliers by calculating the similarity or distance between objects; ensemble-based algorithms combine the outputs of multiple basic anomaly detection models to improve overall performance and robustness; deep learning-based algorithms use multi-layer neural networks to learn the complex features of data and perform anomaly detection on this basis.

[0004] Density Peak Clustering (DPC) is an unsupervised clustering algorithm proposed by Alex Rodriguez and Alessandro Laio in 2014. The algorithm mainly relies on the local density and relative distance of data points to determine the cluster centers, and assigns data points to corresponding clusters based on these centers. Although the DPC algorithm does not require explicit input of parameters such as the number of clusters in theory, in practical applications, its clustering effect will be affected by the distribution of data objects. Summary of the invention

[0005] The technical problems of the present invention are: Existing abnormal data identification algorithms require several input parameters and cannot objectively predict which parameter values ​​are appropriate for unknown data sets; the DPC algorithm does not require any input parameters, but its effect is easily affected by the distribution of data objects.

[0006] The purpose of the present invention is to solve the above problems and provide a new abnormal data identification method based on K-means algorithm; firstly, the data set is divided into data clusters. Since the distribution of abnormal objects is relatively sparse, the abnormal objects will be divided into data clusters containing only a small number of data objects; then, the number of objects in each data cluster is counted, and a probability distribution curve diagram is generated, which is named as a decision diagram; from the decision diagram, the user can clearly identify which clusters have a small number of objects and a low probability, so as to identify them as abnormal clusters, because the number of abnormal data objects is much smaller than that of normal data objects, and the number of objects in the cluster where the abnormal data objects are located is also much smaller than the number of objects in the cluster where the normal data objects are located, and the data objects therein are identified as abnormal.

[0007] The technical solution of the present invention is an abnormal data identification method based on K-means algorithm, comprising the following steps: S1, divide the data set into data clusters; S2. Use the K-outlier algorithm to cluster the data objects, obtain the cluster center points of each data cluster, calculate the distance between all data objects in the data set cluster and each cluster center point, and assign the data objects to the data cluster with the closest cluster center according to the distance; S3, dividing the value range of the data object into multiple intervals, calculating the interval of each interval and the probability of the number of data objects in each interval after the division, and drawing a probability distribution curve according to the probability of the number of data objects, and using the probability distribution curve as a decision diagram; S4. Based on the decision graph, separate the abnormal data objects from the normal data objects, and identify the inflection point area in the decision graph. S5. The data cluster in the inflection point area is determined as an abnormal cluster, and all data objects in the abnormal cluster are abnormal data objects.

[0008] Furthermore, in step S1, the data set X is divided into K data clusters: , ,… ,in, Indicates j data clusters, and K is the number of data clusters.

[0009] Preferably, step S2 specifically includes the following sub-steps: S21, initialization: randomly selecting a data object in each data cluster as the initial clustering center of the data cluster; S22, calculate each data object and each cluster center Distance , the calculation formula is:

[0010] In the formula, is the dataset dimension, represents the i-th object in the dataset, Indicates j Cluster centers, , They are , The value of the tth dimension of ; S23. Recalculate the cluster center of each data cluster according to the assigned data cluster , the mean of all objects in each data cluster is used as the new cluster center, and the calculation formula is:

[0011] in, Indicates the data cluster The number of objects; S24. Repeat steps S22 and S23 until the cluster center of each data cluster no longer changes.

[0012] Preferably, step S2 further includes minimizing the sum of square distances between data points within the cluster and their corresponding cluster centers using an optimization loss function, and the calculation formula is as follows:

[0013] In the formula, x is the data object of the data set X; It is Clusters, is the cluster center of the cluster, Represents data object x to cluster center The square of the Euclidean distance.

[0014] Furthermore, in step S3, the interval The calculation formula is:

[0015] In the formula and They are and The number of objects in Furthermore, in step S3, the tth interval The probability of the number of data objects in The calculation formula is:

[0016] In the formula, θ represents the cluster center Is the function in the interval? like ,but ;otherwise .

[0017] Preferably, the step S5 further comprises: visualizing the number and distribution of abnormal data objects based on the decision diagram to obtain a distribution diagram of the abnormal data objects.

[0018] Compared with the prior art, the beneficial effects of the present invention include: 1. The present invention provides an anomaly identification method based on K-means, which identifies a large number of small clusters through the K-means clustering algorithm, and then counts the distribution probability of the number of objects in each cluster to generate a probability distribution curve. From the probability distribution curve, it can be clearly observed which clusters have significantly fewer objects than other clusters, so that they are identified as abnormal clusters, and the objects in them are identified as abnormal.

[0019] 2. The present invention uses a visual decision diagram instead of input parameters. Unlike other anomaly recognition algorithms, the present invention does not require input parameters to identify anomalies. Users can obtain anomaly recognition results through the visual information provided by the decision diagram.

[0020] 3. The abnormality recognition effect of the present invention is not affected by the distribution of data objects. The abnormal data objects are divided into a small number of data clusters and distinguished from normal data objects, which is not affected by the distribution of normal objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0022] Figure 1 Schematic diagram of an abnormal data identification method according to an embodiment of the present invention.

[0023] Figure 2 4 is a clustering result diagram of four density-imbalanced clusters in an embodiment of the present invention.

[0024] Figure 3 This is a decision diagram generated for four density-imbalanced clusters in an embodiment of the present invention.

[0025] Figure 4 This is a diagram showing the clustering results of the data set aggregation in an embodiment of the present invention.

[0026] Figure 5 It is a decision diagram generated for data set aggregation in an embodiment of the present invention.

[0027] Figure 6 This is a diagram showing the clustering results of the clusto-t4.8k data set in an embodiment of the present invention.

[0028] Figure 7 This is a decision diagram generated for the clusto-t4.8k data set in an embodiment of the present invention.

[0029] Figure 8 This is a clustering result diagram of the clusto-t7.10k data set in an embodiment of the present invention.

[0030] Fig. 9 This is a decision diagram generated for the clusto-t7.10k data set in an embodiment of the present invention.

[0031] Fig.10 This is a diagram showing the clustering results of the clusto-t8.8k data set in an embodiment of the present invention.

[0032] Fig.11 This is a decision diagram generated for the clusto-t8.8k data set in an embodiment of the present invention.

[0033] Fig.12 This is a diagram showing the clustering results of the cure-t2.4k dataset in an embodiment of the present invention.

[0034] Fig.13 This is a decision diagram generated for the cure-t2.4k dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Example 1, reference Figure 1 The present invention provides an abnormal data identification method based on K-means algorithm, comprising the following steps: S1. Divide the data set X into K clusters; S2. Use the K-means algorithm to cluster the data objects, obtain the cluster center points of each data cluster, calculate the distance between all data objects in the data set cluster and each cluster center point, and assign the data objects to the data cluster with the closest cluster center according to the distance; S3, dividing the value range of the data object into multiple intervals, calculating the intervals of each interval and the probability of the number of data objects in each interval after the division, and drawing a probability distribution curve according to the probability of the number of data objects, and using the curve as a decision diagram; S4. Based on the decision graph, the abnormal data objects are isolated from the normal data objects, and the inflection point area in the decision graph is identified; S5. The data cluster in the inflection point area is determined as an abnormal cluster, and all data objects in the abnormal cluster are abnormal.

[0037] Furthermore, in step S1, the data set X is divided into K data clusters: , ,… ,in, Indicates j data clusters, and K is the number of data clusters.

[0038] Furthermore, step S2 includes the following sub-steps: S21, initialization: randomly selecting a data object in each data cluster as the initial clustering center of the data cluster; S22, calculate each data object and each cluster center Distance , the calculation formula is:

[0039] In the formula, is the dataset dimension, represents the i-th object in the dataset, represents the jth cluster center, , They are , The value of the tth dimension of ; S23. Recalculate the cluster center of each data cluster according to the assigned data cluster , the mean of all objects in each data cluster is used as the new cluster center, and the calculation formula is:

[0040] in, Indicates the data cluster The number of objects; S24. Repeat steps S22 and S23 until the cluster center of each data cluster no longer changes.

[0041] Furthermore, step S2 also includes minimizing the sum of square distances between data points in the cluster and their corresponding cluster centers by optimizing the loss function, and the calculation formula is as follows:

[0042] In the formula, x=( ), is the data object of data set x; It is Clusters, is the cluster center of the cluster, Represents data object x to cluster center The square of the Euclidean distance.

[0043] In step S3 of the embodiment, compared with normal data objects, abnormal data objects are sparsely distributed and are only contained in small clusters of a small number of objects.

[0044] Furthermore, step S4 includes the following sub-steps: S41: Divide the value range of the number of objects into intervals, and the interval of each interval is , the calculation formula is:

[0045] in and They are and The number of objects in S42: Calculate the probability of the number of objects in each interval. For the tth interval The probability of the number of data objects in The calculation formula is:

[0046] In the formula, θ represents the cluster center Is it within the interval? like ,but ;otherwise .

[0047] In step S5 of the embodiment, abnormal objects are isolated from normal objects, and the abnormal identification effect is not affected by the distribution of objects; In an embodiment, the K-means-based anomaly identification method further includes: visualizing the number and distribution of abnormal data objects based on a decision diagram to obtain a distribution diagram of the abnormal data objects.

[0048] The K-outlier algorithm has certain robustness to density-imbalanced data sets and complex distribution data. The method of the present invention was tested on a data set containing 4 density-imbalanced clusters. The experimental results and decision diagram are shown in Figure 2 and Figure 3 shown.

[0049] In the embodiment, the method of the present invention is tested on 5 data sets, namely aggregation, clusto-t4.8k, clusto-t7.10k, clusto-t8.8k, cure-t2.4k. The experimental results and decision diagram are shown in Figure 3-Figure 13 shown.

[0050] As can be seen from the figure, No matter what distribution the data set follows or what shape clusters it forms, the decision graph of the present invention shows obvious inflection points, and most of the abnormal objects are identified. Obviously, the method of the present invention is not greatly affected by the distribution of objects. The effectiveness of the method of the present invention is verified by experiments, indicating that the present invention is robust to density-imbalanced data sets, complex distribution data, and complex data object distribution.

[0051] In addition to the robustness experiment, the method of the present invention is compared with other algorithms on 7 real-world data sets. These algorithms are ABOD, CBLOF, COF, Isolation Forest, INNE, KNN, and KPCA. The evaluation index is F1. The experimental results are shown in Table 1.

[0052] Table 1 Accuracy comparison between the present invention and the comparative algorithm

[0053] The experimental results show that the method of the present invention achieves the best results on all data sets. Therefore, the method of the present invention is a very effective abnormal data identification algorithm.

[0054] The implementation results show that the main contributions of the anomaly recognition algorithm of the present invention include: 1. Generate a visual decision diagram instead of input parameters: Unlike other anomaly recognition algorithms, the method of the present invention no longer requires input parameters to identify anomalies. Users can directly obtain anomaly recognition results through the visual information provided by the decision diagram; 2. The anomaly recognition effect of the method of the present invention is not affected by the distribution of objects; since the anomalies are divided into a small number of data clusters and separated from normal objects, they are not affected by the distribution of normal objects.

[0055] 3. The method of the present invention is robust to density-imbalanced data sets and complex distribution data sets.

Claims

1. A method for identifying abnormal data based on K-means algorithm, characterized in that: The following steps are involved: S1, divide the data set into data clusters; S2. Use the K-means algorithm to cluster the data set, obtain the cluster center of each data cluster, calculate the distance between all data objects in the data set and each cluster center, and assign the data object to the data cluster with the closest cluster center based on the distance; S3, dividing the value range of the data object into multiple intervals, and calculating the interval of each interval and the probability of the number of data objects in each interval, and drawing a probability distribution curve according to the probability of the number of data objects, and using the probability distribution curve as a decision diagram; S4. Based on the decision graph, separate the abnormal data objects from the normal data objects, and identify the inflection point area in the decision graph; S5. The data cluster in the inflection point region is determined to be an abnormal cluster, and all data objects in the abnormal cluster are abnormal data objects.

2. The abnormal data identification method based on the K-means algorithm according to claim 1 is characterized in that: In step S1, the data set X is divided into K data clusters: , ,… ,in, represents the jth data cluster, and K is the number of data clusters.

3. The abnormal data identification method based on K-means algorithm according to claim 2 is characterized in that: Step S2 specifically includes the following sub-steps: S21, initialization: randomly selecting a data object in each data cluster as the initial clustering center of the data cluster; S22, calculate each data object and each cluster center Distance , the calculation formula is: In the formula, is the dataset dimension, represents the i-th object in the dataset, Indicates j Cluster centers, , They are , The value of the tth dimension of ; S23. Recalculate the cluster center of each data cluster according to the assigned data cluster , the mean of all objects in each data cluster is used as the new cluster center, and the calculation formula is: in, Indicates the data cluster The number of objects; S24, repeating step S22 and step S23 until the cluster center of each data cluster no longer changes.

4. The abnormal data identification method based on K-means algorithm according to claim 3 is characterized in that: The step S2 further includes: minimizing the sum of the square distances between the data objects in the data cluster and their corresponding cluster centers by using an optimization loss function, the calculation formula being: In the formula, x is the data object of dataset X; It is Data clusters, is the cluster center of the data cluster, Represents a data object x To cluster center The square of the Euclidean distance.

5. The abnormal data identification method based on K-means algorithm according to claim 4 is characterized in that: In step S3, the value range of the data object is divided into equal intervals, and the intervals are The calculation formula is: In the formula and They are and The number of data objects in .

6. The abnormal data identification method based on K-means algorithm according to claim 5 is characterized in that: In step S3, the yth interval The probability of the number of data objects in The calculation formula is: In the formula, θ represents the cluster center Is the function in the interval? like ,but ;otherwise .

7. The abnormal data identification method based on K-means algorithm according to claim 6 is characterized in that: The step S5 further includes: visualizing the number and distribution of abnormal data objects based on the decision diagram to obtain a distribution diagram of the abnormal data objects.

Images