Multidimensional anomaly detection model based on spatial density

Abstract

Multidimensional anomaly detection model based on spatial density. The present invention is an anomaly detection model that creates a data space, calculates the circle value of each data point, thereby obtaining a critical value, and finally uses the critical value to determine whether the data point is abnormal or not. Because a pure mathematical method is used to implement the anomaly detection function and to minimize the intervention of artificial rules, this anomaly detection model can be widely used in various occasions, such as product quality inspection, network anomaly data mining, equipment failure warning, etc. The present invention mainly includes three parts: feature engineering, anomaly detection model training, and anomaly detection model detection. In particular, the anomaly detection model training stage creatively solves the overlapping and edge problems that other anomaly detection methods cannot handle, thereby achieving a very high detection accuracy.

Description

Technical Field

[0001] The present invention relates to computer technology and big data, and in particular to the discovery and processing of abnormal data. Background Art

[0002] In big data analysis and processing technology, abnormal data detection is a very critical technology, which is widely used in industry, medical care, finance, Internet and other fields, and plays an important role in abnormal data capture, abnormal behavior discovery, fault warning and other aspects.

[0003] Currently commonly used anomaly detection technologies include: rule-based anomaly detection technology; statistics-based anomaly detection technology; clustering-based anomaly detection technology; machine learning-based anomaly detection technology, etc.

[0004] Taking the commonly used independent forest anomaly detection algorithm as an example, it continuously splits data points so that each data point obtains the fastest independent splitting path, and the degree of anomaly is determined by the speed of data point independence. However, it does not have a good solution to the overlap problem and edge problem. In order to solve the defects of these commonly used algorithms, the significance of the present invention is to explore a universal anomaly detection model without obvious loopholes and human intervention. Summary of the invention

[0005] The present invention is a multi-dimensional anomaly detection algorithm based on spatial density. Its core idea is that normal data must be gregarious and clustered with most other data, while abnormal data is not gregarious and is discrete, so the data around the abnormal data must be relatively small. Based on this idea, we can calculate the data density around each data point, and thus infer whether this data point is abnormal.

[0006] This anomaly detection algorithm supports multi-dimensional data, which makes it very versatile. When using this model, users only need to convert the data dimensions that need to be detected into features and pass them into the model. They do not need to worry about the calculation process. The model can automatically identify the abnormal data. This will make anomaly detection very simple in applications with very complex data dimensions.

[0007] The present invention is mainly divided into three stages: feature engineering, model training, and model testing.

[0008] Feature Engineering: Because this anomaly detection model has strict requirements on the input data, the validity and correctness of the data set must be guaranteed. First, the size of the data set must reach a certain level, because if the data is too small, the sample space will show limitations, which will eventually lead to deviations in the detection results. Secondly, try to delete invalid features that have no effect on the results. Invalid features will affect the spatial density of data points in the data space. Since these features are invalid, just clean them up. Finally, clean up the contradictory data. Because of errors, sampling failures, etc., similar data may result in different results. For such data, first analyze which data is correct and which data is wrong, delete the wrong data, and keep the correct data, or directly delete all contradictory data without affecting the amount of data. When the data is processed, we will get a valid data set. Finally, turn this data set into a digital vector, similar to: [[x11,x12…], [x21,x22…], [x31,x32…]…] In this way, the outer brackets represent the entire data set, the inner brackets represent a data point, and x represents the characteristics of the data point.

[0009] Model training: The purpose of model training is to obtain a critical value. After obtaining the critical value, you can choose to save the model to the hard disk. The next time you encounter the same data, you can directly read the model without training the model again, and you can detect anomalies in such data.

[0010] Model training step 1): Data normalization. In order to ensure that the value range of each dimension in the data space is uniform, it is necessary to normalize the data set before training. First, we save the original data in the model class, then calculate the upper and lower limits of each dimension, and finally change the eigenvalue of each dimension to the interval [0,1]. The specific calculation formula is , x(new): new data value, x(old): old data value, x(max): current dimension data maximum value, x(min): current dimension data minimum value. We also need to save the upper and lower limits of each dimension to the model class, which is to facilitate the normalization of the detection data later.

[0011] Model training step 2): After completing data normalization, we get a standard data space with each dimension length of 1. Let the radius R = 0.1, take the data point as the center of the circle, R as the radius, and calculate the circle value of the current data point.

[0012] Model training step 3): When the target data point is at the edge of the data space, drawing a circle around this data point will result in an out-of-bounds problem, and the statistical circle value is incorrect. This is the edge problem. In order to solve the edge problem, we can add a weight value to these data points at the edge of the data space, and finally multiply the statistical circle value by the weight value. The calculation formula for the weight value is: , n: data dimension, R: circle radius, D: the distance from the data point to the edge of the current dimension (divided into left and right sides).

[0013] Model training step 4): After obtaining the circle values ​​of all data points in the training data set, the critical value can be calculated by the formula. The specific formula is: , w: tightness, Qi: circle value, n: number of data points. The w tightness represents the strictness of abnormality or not, because everyone has a different definition of abnormality, some people have strict requirements for abnormality, and some people have loose requirements. The w value is used to adjust this strictness. Simply put, the larger the w value, the looser the abnormality requirement (the more abnormalities), and the smaller the w value, the stricter the abnormality requirement (the fewer abnormalities).

[0014] Model detection: After the anomaly detection model training is completed, the attributes contained in a model object include the original data set, the upper and lower limits of each dimension, and the critical value. We can save the model object to the specified hard disk space for easy use next time. When you need to detect whether a data point of the same type is abnormal, first restore the previously saved model object, then call the detection function to pass the data point as a parameter. The model will first normalize the data point according to the saved dimension upper and lower limit information, then calculate the circle value of this data point, and then compare this circle value with the critical value, and finally get the conclusion whether this data point is abnormal. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a computational flow chart of a multidimensional anomaly detection model based on spatial density provided in an embodiment of the present application; DETAILED DESCRIPTION

[0016] Feature Engineering:

[0017] Step 1) Collect and summarize raw data, which can be probe information, measurement information, log information, observation information, etc.

[0018] Step 2) Arrange the original data into uniform dimension lengths to ensure that each dimension data has the same meaning;

[0019] Step 3) Remove dimensions that have no effect on the results;

[0020] Step 4) Remove contradictory data;

[0021] Step 5) Digitize the raw data;

[0022] Step 6) Integrate into a unified data structure: [[x11,x12…],[x21,x22…],[x31,x32…]…].

[0023] Model training and testing phase:

[0024] Step 1) Create an anomaly detection model instance object and determine the value of tightness w;

[0025] Step 2) Pass the training data into the model object;

[0026] Step 3) Normalize the data and save the upper and lower limit information of each dimension;

[0027] Step 4) Calculate the critical value and save the critical value information;

[0028] Step 5) Save the model for next use;

[0029] Step 6) Restore the model, pass the detection data into the model, and obtain the anomaly detection results.

Claims

1. A spatial density-based anomaly detection method, characterized in that: include: S10, obtaining a data set, the original data of the data set is probe information; removing features that have no effect on the results; And clean up the contradictory data; convert the data set into the form of digital vectors; The data points and their features are included in the form of numeric vectors; S20, training of anomaly detection model, wherein the training of anomaly detection model is to obtain a critical value. After obtaining the critical value, the model is saved to the hard disk. Next time the same type of data is encountered, the model is directly read to detect anomalies of such data; including: S201, data normalization: normalize the data set before training; calculate the upper and lower limits of each dimension, and finally change the eigenvalue of each dimension to the interval [0,1]; S202, model training: After completing data normalization, a standard data space with a length of 1 in each dimension is obtained, and the radius is set to R. With the data point as the center of the circle, the number of data points in the multidimensional space whose distance from the center of the circle does not exceed the radius is counted, and the circle value of the current data point is calculated; When the target data point is at the edge of the data space, a weight value is added to the data points at the edge of the data space, and finally the circle value is multiplied by the weight value; The calculation formula of the weight value is: Among them, n is the data dimension, R is the circle radius, and D li is the distance from the data point to the left of the current dimension, D ri is the distance from the data point to the right of the current dimension; W is the weight value; After obtaining the circle values ​​of all data points in the data set, the critical value is calculated by the following formula; Among them, w is the tightness, Qi is the circle value, m is the number of data points; V is the critical value; S30, model detection: after obtaining the critical value, the anomaly detection model is saved to the specified hard disk space; when it is necessary to detect whether a data point of the same type is abnormal, the saved model object is restored, and then the detection function is called to pass the data point as a parameter into the model. The model will first normalize the data point according to the saved dimension upper and lower limit information, and then calculate the circle value of the data point, and then compare the circle value with the critical value, and finally get the conclusion whether the data point is abnormal.

2. The method for detecting anomalies based on spatial density according to claim 1, characterized in that: R＝0.1。

Images