An abnormal data detection technology based on grid classification

Abstract

The present invention belongs to the technical field of computer equipment, and in particular, is a kind of abnormal data detection technology based on grid classification. Aiming at the problem that iForest has certain randomness in classification, clustering and abnormality detection, which leads to reduced algorithm reliability, and iforest is sensitive to global sparse points and is not good at processing local relatively sparse points, the following scheme is proposed, including the following steps, recording the number of data as n, performing PCA dimension reduction on the data, recording the number of all data as n, using the grid to divide the data space after dimension reduction into several regions, the number of regions is recorded as f, and eliminating the empty grid region. The present invention performs iForest abnormality detection operation on the grid classified region, finds outliers, then enlarges the spatial region, performs secondary iForest classification on the region where the outliers are located, compares the two outliers, and outputs them as outliers if they are the same, first performs grid classification, reduces the randomness of iForest selecting samples, and makes the abnormality classification more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of computer devices, and particularly to an abnormal data detection technology based on grid classification. Background Art

[0002] With the rapid development of Internet information technology, various data are constantly collected and stored in databases. It is very important to mine some helpful information from these scattered data. However, there will be some abnormal data in these data, and the correction of abnormal data has gradually attracted the attention of researchers and become an important part of the data mining field.

[0003] Abnormal data will have a greater impact on the results of data processing, especially interfering with the finally output data. iForest has achieved remarkable results in classification clustering and anomaly detection. However, since iForest randomly selects a dimension each time it cuts the data space, which is random, and there is still a large amount of dimension information not used after building the tree, resulting in a reduction in the reliability of the algorithm and causing certain difficulties in correcting abnormal data. Summary of the Invention

[0004] Based on the technical problems that iForest has certain randomness in classification clustering and anomaly detection, resulting in a reduction in algorithm reliability, and iForest is sensitive to global sparse points and not good at dealing with local relatively sparse points, the present invention proposes an abnormal data detection technology based on grid classification.

[0005] An abnormal data detection technology based on grid classification proposed by the present invention includes the following steps:

[0006] S1: Record the number of data as n, perform PCA dimensionality reduction on the data, and record the number of all data as n;

[0007] S2: Use a grid to divide the dimensionality-reduced data space into several regions, record the number of regions as f,剔除 empty grid regions, record the number of remaining grid regions as N, calculate the local density p within each grid, and the local density is represented by the number of points. If p < n / N / 6, all are regarded as abnormal points;

[0008] S3: Perform iForest anomaly detection operations on each non-empty region, initially screen out abnormal points. If the number of samples in a certain region is much smaller than the average value, or if the basic layer classification cannot be completed during iForest classification, then mark the points in this region as abnormal points;

[0009] S4: Enlarge the spatial region, with the grid region side = 2side, then perform iForest classification on the region where the anomaly points are located, and compare the obtained anomaly points with the previous anomaly points. If they are the same, output them as anomaly points.

[0010] Preferably, in the S1, through PCA dimensionality reduction, map the n-dimensional features onto the k-dimensional space (k < n), reduce the interference of non-related features, and retain the features with a larger correlation coefficient with the anomaly points to obtain the principal components of the data.

[0011] Preferably, in the S2, calculate the grid classification parameter side of the d-dimensional dataset, divide the data space into equal and non-overlapping grid cells, and the grid side length side is determined by the specific data volume and should not be too large;

[0012] Assume that the center point of grid j is j = 1, 2,..., K, where K is the total number of grids, and represents the coordinates of the center point of grid j in the d dimension. Therefore, grid G j can be expressed as:

[0013]

[0014] The set of points in grid j is P = {p1, p2,..., p lj}, and l j is the total number of data points in this grid j. Then the representative point of this grid is:

[0015]

[0016] The local density of the grid representative point P j is represented by the number of data points in grid j. The number of points in grid j is:

[0017]

[0018]

[0019] Then the local density of the grid representative point P j is ρ j = l j ;

[0020] Take the closest distance to the higher density representative point P j of the grid representative point P k as the distance value of the grid representative point P j , denoted as δ j :

[0021]

[0022] where Dkj is the grid representative point P k With the grid representative point P j distance;

[0023]

[0024]

[0025] When the grid representative point object meets the above two formula conditions, the grid representative point is selected as the cluster center, where is the density of the cluster center grid representative points, μ(ρ i ) is the mean value of the density of all grid representative points, is the minimum distance between the grid representative point and the cluster center representative point in the same cluster, E(δ i ) is all δ i expectations.

[0026] Preferably, in S3, the data is segmented by a binary tree until the data cannot be divided any further or a specified number of iterations is reached;

[0027] First, follow a tree itree, starting from the root node and going down according to the values ​​of different features until reaching a leaf node. Assuming that the number of samples in the training samples of itree that also fall on the leaf node where x is located is T.size, then the path length of data x on this tree itree is h(x), which can be calculated using the following formula:

[0028] h(x)=e+C(T.size)

[0029] Where e represents the number of edges that data x passes through from the root node to the leaf node of itree, and C(T.size) can be considered as a correction value, which represents the average path length of a binary tree constructed with T.size sample data:

[0030] C(n)=2H(n-1)-2(n-1) / n

[0031] Among them, H(n-1) can be estimated by ln(n-1)+0.5772156649, where the constant is Euler's constant;

[0032] The final anomaly score of data x combines the results of multiple trees:

[0033]

[0034] If the average path length of data x in multiple trees is shorter, the score is closer to 1, indicating that the data x is more abnormal. If the average path length of data x in multiple trees is longer, the score is closer to 0, indicating that the data x is more normal.

[0035] The beneficial effects of the present invention are as follows:

[0036] By performing iForest anomaly detection operations on the regions classified by the grid, the anomaly points are found, then the spatial region is enlarged, and a secondary iForest classification is performed on the region where the anomaly points are located. The anomaly points of the two times are compared. If they are the same, they are output as anomaly points. First, grid classification is performed to reduce the randomness of sample selection by iForest, making the anomaly classification more accurate. Specific embodiments

[0037] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments.

[0038] An anomaly data detection technology based on grid classification includes the following steps:

[0039] S1: Record the number of data as n, perform PCA dimensionality reduction on the data, and record the number of all data as n;

[0040] S2: Use the grid to divide the dimensionality-reduced data space into several regions. The number of regions is denoted as f. Eliminate the empty grid regions, and denote the number of remaining grid regions as N. Calculate the local density p in each grid. The local density is represented by the number of points. If p < n / N / 6, all are regarded as anomaly points;

[0041] S3: Perform iForest anomaly detection operations on each non-empty region, and initially screen out the anomaly points. If the number of samples in a certain region is much smaller than the average value, or the basic layer classification cannot be completed when performing iForest classification, then the points in this region are marked as anomaly points;

[0042] S4: Enlarge the spatial region, the side of the grid region is side = 2side, and then perform iForest classification on the region where the anomaly points are located. The obtained anomaly points are compared with the previous anomaly points. If they are the same, they are output as anomaly points. The purpose of the second classification here is to greatly reduce the false alarm anomalies.

[0043] In S1, through PCA dimensionality reduction, the n-dimensional features are mapped to k dimensions (k < n), reducing the interference of non-correlated features and retaining the features with a larger correlation coefficient with the anomaly points to obtain the principal components of the data. Since iforest is not applicable to particularly high dimensions, the dimensionality-reduced data should be controlled within a lower dimension.

[0044] In S2, the grid classification parameter side of the d-dimensional data set is calculated to divide the data space into equal and non-intersecting grid units. The grid side length side is determined by the specific data volume and should not be too large.

[0045] Assume that the center point of grid j is j = 1, 2, ..., K, where K is the total number of grids. represents the coordinates of the center point of the grid j in d dimensions, so the grid G j It can be expressed as:

[0046]

[0047] The set of points in grid j is P = {p1, p2, ..., p lj}, l j is the total number of data points in the grid j, then the representative point of the grid is:

[0048]

[0049] Grid representative point P j The local density of is expressed by the number of data points in grid j, and the number of points in grid j is:

[0050]

[0051]

[0052] Then the grid represents point P j The local density is ρ j = l j ;

[0053] Representing point P with a grid j Higher density represents point P k The closest distance, as the grid representative point P j The distance value is denoted as δ j :

[0054]

[0055] Where D kj is the grid representative point P k With the grid representative point P j distance;

[0056]

[0057]

[0058] When the grid representative point object meets the above two formula conditions, the grid representative point is selected as the cluster center, where is the density of the cluster center grid representative points, μ(ρ i ) is the mean value of the density of all grid representative points, is the minimum distance between the grid representative point and the cluster center representative point in the same cluster, E(δ i ) is all δ i expectations.

[0059] In S3, the data is segmented by a binary tree until the data cannot be divided any further or the specified number of iterations is reached;

[0060] First, follow a tree itree, starting from the root node and going down according to the values ​​of different features until reaching a leaf node. Assuming that the number of samples in the training samples of itree that also fall on the leaf node where x is located is T.size, then the path length of data x on this tree itree is h(x), which can be calculated using the following formula:

[0061] h(x)=e+C(T.size)

[0062] Where e represents the number of edges that data x passes through from the root node to the leaf node of itree, and C(T.size) can be considered as a correction value, which represents the average path length of a binary tree constructed with T.size sample data:

[0063] C(n)=2H(n-1)-2(n-1) / n

[0064] Among them, H(n-1) can be estimated by ln(n-1)+0.5772156649, where the constant is Euler's constant;

[0065] The final anomaly score of data x combines the results of multiple trees:

[0066]

[0067] If the average path length of data x in multiple trees is shorter, the score is closer to 1, indicating that the data x is more abnormal. If the average path length of data x in multiple trees is longer, the score is closer to 0, indicating that the data x is more normal.

[0068] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An abnormal data detection technology based on grid classification, characterized in that: It includes the following steps: S1: Record the number of data as n, perform PCA dimensionality reduction on the data, and record the number of all data as n; S2: Use a grid to divide the dimensionality-reduced data space into several regions. The number of regions is denoted as f. Eliminate the empty grid regions, and denote the number of remaining grid regions as N. Calculate the local density p within each grid. The local density is represented by the number of points. If p < n / N / 6, all are regarded as outliers; S3: Perform iForest outlier detection operations on each non-empty region, and initially screen out outliers. If the number of samples in a certain region is much smaller than the average value, or if the basic hierarchical classification cannot be completed during iForest classification, then mark the points in this region as outliers; S4: Enlarge the spatial region, set the grid region side = 2side, and then perform iForest classification on the region where the outliers are located. Compare the obtained outliers with the previous outliers. If they are the same, output them as outliers.

2. The abnormal data detection technology based on grid classification according to claim 1 is characterized in that: In S1, through PCA dimensionality reduction, map the n-dimensional features to k dimensions (k < n), reduce the interference of non-correlated features, and retain the features with a relatively large correlation coefficient with the outliers to obtain the principal components of the data.

3. The abnormal data detection technology based on grid classification according to claim 1 is characterized in that: In S2, calculate the grid classification parameter side of the d-dimensional dataset, and divide the data space into equal and non-overlapping grid cells. The grid side length side depends on the specific data volume and should not be too large; Assume that the center point of grid j is K is the total number of grids, where represents the coordinates of the center point of the grid j in d dimensions, so the grid G j It can be expressed as: The set of points in grid j is P = {p1, p2, ..., p lj }, l j is the total number of data points in the grid j, then the representative point of the grid is: Grid representative point P j The local density of is expressed by the number of data points in grid j, and the number of points in grid j is: Then the grid represents point P j The local density is ρ j = l j ; Represent point P with a grid j Higher density represents point P k The closest distance, as the grid representative point P j The distance value is denoted as δ j : Where D kj is the grid representative point P k With the grid representative point P j distance; When the grid representative point object meets the above two formula conditions, the grid representative point is selected as the cluster center, where is the density of the cluster center grid representative points, μ(ρ i ) is the mean value of the density of all grid representative points, is the minimum distance between the grid representative point and the cluster center representative point in the same cluster, E(δ i ) is all δ i expectations.

4. The abnormal data detection technology based on grid classification according to claim 1 is characterized in that: In S3, divide the data through a binary tree until the data cannot be further divided or the specified number of iterations is reached; First, along a tree itree, starting from the root node, go down according to the values of different features until reaching a certain leaf node. Suppose the number of samples in the training samples of itree that also fall in the leaf node where x is located is T.size. Then the path length of the data x on this tree itree is h(x), which can be calculated by the following formula: h(x) = e + C(T.size) where e represents the number of edges that the data x experiences from the root node to the leaf node of itree, and C(T.size) can be regarded as a correction value, representing the average path length of a binary tree constructed with T.size sample data: C(n) = 2H(n - 1) - 2(n - 1) / n where H(n - 1) can be estimated by ln(n - 1) + 0.5772156649, and the constant here is the Euler constant; The final outlier score of the data x synthesizes the results of multiple trees: If the average path length of the data x in multiple trees is shorter, the score is closer to 1, indicating that the data x is more abnormal. If the average path length of the data x in multiple trees is longer, the score is closer to 0, indicating that the data x is more normal.

Images