Unsupervised anomaly detection method and device based on local area angle information, and medium

Abstract

The invention relates to the technical field of data anomaly detection, in particular to an unsupervised anomaly detection method and device based on local area angle information. According to the method, the concept of localization is introduced, the local influence set of each data point is determined firstly, the influence set is composed of the nearest neighbor set and the reverse nearest neighbor set of the data point, and therefore the calculation range is limited, and the calculation complexity is reduced. Then, the variance of angles between the difference vector from the target data point to the mean value of the influence set and the difference vector from the target point to each data point in the influence set is innovatively calculated, and the variance is used as an index for evaluating the local anomaly degree of the data points. The variance is defined based on the angle change in the local area, so that the abnormal characteristics of the data local structure can be carefully captured, and the limitation of the ABOD method on a multi-modal data set is effectively overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of data anomaly detection, and in particular to an unsupervised anomaly detection method and device based on local area angle information. Background Art

[0002] In the field of data mining, outlier detection, as a core task, has been widely used in many key fields, such as email spam detection, network security protection (including intrusion detection and anomaly recognition), medical diagnosis assistance, malware identification, anomaly recognition in video and hyperspectral image analysis, and damage detection in the industrial field. Outlier detection aims to identify abnormal data points from the data set that are significantly inconsistent with the main distribution. These abnormal points often contain important information, such as critically ill patients in medicine or emergency events in intelligent transportation systems.

[0003] According to the availability of label information, existing anomaly detection technologies can be roughly divided into three categories: unsupervised, semi-supervised, and supervised. Supervised learning methods, also known as supervised anomaly detection, rely on training data of known categories. Although they are highly accurate, the acquisition of labeled data is expensive and time-consuming. Semi-supervised methods are designed for situations where labels are scarce and are suitable for scenarios where a small number of positive examples and a large amount of unlabeled data coexist. In contrast, unsupervised anomaly detection methods do not require any pre-labeled data sets, so they show a wider range of application potential. Among them, the unsupervised anomaly detection method based on the nearest neighbor has gradually attracted widespread attention from researchers because of its feature that it does not require any prior distribution assumptions. Among the nearest neighbor-based anomaly detection methods, they can be mainly divided into two categories: distance-based methods and density-based methods. Distance-based methods calculate the distance from an instance to its first The distance or sum of distances of the nearest neighbors is used to evaluate the abnormality of the instance. As a preset parameter, this method ignores the differences between instances with the same distance but different densities. To solve this problem, density-based methods have emerged, among which the most representative one is the Local Outlier Factor (LOF) algorithm. The LOF algorithm compares the local density of an instance with its The average local density of the nearest neighbors is used to assign an anomaly score to each instance, thereby achieving more accurate anomaly detection.

[0004] Although the LOF algorithm and its improved methods have made remarkable achievements in the field of anomaly detection, they still face many challenges when facing high-dimensional data and complex distributed data sets. For example, angle information, as a feature that is insensitive to the increase in dimensionality, has been introduced into the field of anomaly detection, and the angle-based outlier detection (ABOD) method has been proposed. The ABOD method identifies outliers by calculating the angular variance of the difference vector between the instance and the remaining instances, effectively alleviating the "distance concentration effect" problem. However, this method has a high computational time complexity (O(n^3)), where n represents the number of instances, and only uses global angle information, and often performs poorly in multimodal data sets.

[0005] With the advent of the big data era, anomaly detection of high-dimensional and multimodal data has become an important research topic. Traditional distance-related or density-related anomaly detectors often face the problem of "dimensionality curse" when processing such data, that is, as the data dimension increases, the computational complexity and detection performance drop sharply.

[0006] To overcome this problem, the Angle-Based Outlier Detection (ABOD) method was proposed, which identifies outliers by calculating the angle of the difference vector between data points, thereby alleviating the impact of the dimensionality curse to a certain extent. However, the performance of the ABOD method on multimodal datasets is not satisfactory because it mainly relies on global information and has difficulty capturing the complexity of the local structure of the data. In addition, the ABOD method needs to calculate the angle of the difference vector between each pair of instances in the dataset, which leads to extremely high time complexity and limits its promotion in practical applications. Summary of the invention

[0007] The present invention aims to solve the above-mentioned technical bottlenecks, and proposes a more efficient and accurate anomaly detection method by introducing the concept of localization. Specifically, the present invention first determines the local influence set of each data point, which is composed of the nearest neighbor set and the reverse nearest neighbor set of the data point, thereby limiting the calculation range and reducing the calculation complexity. Subsequently, the present invention innovatively calculates the variance of the angle between the difference vector from the target data point to the mean of its influence set and the difference vector from the target point to each data point in its influence set, and uses this as an indicator to evaluate the degree of local anomaly of the data point. Since the variance is defined based on the angle change in the local area, the method of the present invention can capture the abnormal characteristics of the local structure of the data more finely, effectively overcoming the limitations of the ABOD method on multimodal data sets.

[0008] The present invention provides an unsupervised anomaly detection method based on local area angle information, comprising the following steps:

[0009] S1: Receive a dataset D of outliers to be detected, where D is expressed as:

[0010] ;

[0011] in, represents the pth instance in D;

[0012] n represents the total number of instances in D;

[0013] d represents the dimension of the instance in D;

[0014] S2: Calculate the anomaly factor of each instance in D, where the anomaly factor of the pth instance in D is calculated as:

[0015] S2.1: Calculation of Nearest Neighbor Set ,in, Indicates distance Recent A collection of instances, is the preset parameter;

[0016] S2.2: Calculation Inverse Nearest Neighbor Set , the calculation method is:

[0017] ;

[0018] in, is the qth instance in D, ;

[0019] S2.3: Calculation The impact set , the calculation method is:

[0020] ;

[0021] S2.4: Calculation The abnormal factor , the calculation method is:

[0022] ;

[0023] in, express The average of the midpoints is calculated as:

[0024] ;

[0025] S3: The larger the value, the The higher the probability of being an outlier, the higher the probability of being an outlier. According to the numerical value of the outlier factor of each instance in D, the instance identified as an outlier is given.

[0026] Preferably, the specific method for identifying an instance as abnormal is: according to the numerical value of the abnormal factor, all instances in D are arranged in descending order, and the first r instances are taken as abnormal instances, where r is a preset abnormal value number parameter.

[0027] Preferably, it is characterized in that the calculation When , the distance is expressed as Euclidean distance.

[0028] The present invention also provides an unsupervised anomaly detection device based on local area angle information, comprising the following modules:

[0029] Data receiving module: used to receive the data set D of the abnormal points to be detected, where D is expressed as:

[0030] ;

[0031] in, represents the pth instance in D;

[0032] n represents the total number of instances in D;

[0033] d represents the dimension of the instance in D;

[0034] Anomaly calculation module: used to calculate the anomaly factor of each instance in D, where the calculation method of the anomaly factor of the pth instance in D is:

[0035] S2.1: Calculation of Nearest Neighbor Set ,in, Indicates distance Recent A collection of instances, is the preset parameter;

[0036] S2.2: Calculation Inverse Nearest Neighbor Set , the calculation method is:

[0037] ;

[0038] in, is the qth instance in D, ;

[0039] S2.3: Calculation The impact set , the calculation method is:

[0040] ;

[0041] S2.4: Calculation The abnormal factor , the calculation method is:

[0042] ;

[0043] in, express The average of the midpoints is calculated as:

[0044] ;

[0045] Output module: used to give the instances identified as abnormal according to the numerical value of the abnormal factor of each instance in D; The larger the value, the The higher the probability of being an outlier.

[0046] Preferably, the specific method for identifying an instance as abnormal is: according to the numerical value of the abnormal factor, all instances in D are arranged in descending order, and the first r instances are taken as abnormal instances, where r is a preset abnormal value number parameter.

[0047] Preferably, calculate When , the distance is expressed as Euclidean distance.

[0048] The present invention also provides a computer-readable storage medium,

[0049] The computer-readable storage medium stores a computer program, and the computer program implements the above method steps when executed by a processor.

[0050] Beneficial effects:

[0051] The technical solution of the present invention not only improves the accuracy and efficiency of anomaly detection, but also broadens the application scope of the angle-based anomaly detection method, making it better adapted to the processing requirements of high-dimensional data and multimodal data. Therefore, the present invention has broad application prospects in multiple fields that require data anomaly detection, such as financial risk control, medical diagnosis, and network security. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is an example of a data set in the embodiment;

[0053] Figure 2 In the embodiment In the cluster Schematic diagram of

[0054] Figure 3 In the embodiment At the cluster boundary Schematic diagram of

[0055] Figure 4 In the embodiment When outside the cluster Schematic diagram of

[0056] Figure 5 In the embodiment In the cluster Schematic diagram of

[0057] Figure 6 In the embodiment At the cluster boundary Schematic diagram of

[0058] Figure 7 In the embodiment When outside the cluster Schematic diagram of

[0059] Figure 8 is a Gaussian-distributed artificial data set;

[0060] Fig. 9 It is an artificially synthesized dataset of double Gaussian distribution with overlapping areas;

[0061] Fig.10 It is an artificially synthesized dataset of double Gaussian distribution without overlapping areas;

[0062] Fig.11 The experimental results of ABOD on artificial synthetic data sets;

[0063] Fig.12 The experimental results of FastABOD on artificial synthetic datasets;

[0064] Fig.13 The experimental results of LABOD on artificial synthetic data sets;

[0065] Fig.14 Comparison of the performance (AUC) of ABOD, FastABOD and LABOD on bi-Gaussian distributions of different dimensions;

[0066] Fig.15 Comparison of the performance of ABOD, FastABOD and LABOD on double Gaussian distributions of different dimensions (P@n);

[0067] Fig.16 Comparison of the performance (AP) of ABOD, FastABOD and LABOD on double Gaussian distributions of different dimensions;

[0068] Fig.17 ROC curve for the benchmark dataset (HeartDisease);

[0069] Fig.18 ROC curve for the benchmark dataset (Ionosphere);

[0070] Fig.19 ROC curve for the benchmark dataset (Arrhythmia);

[0071] Fig. 20 ROC curve for the benchmark dataset (Shuttle);

[0072] Fig.21 ROC curve (Waveform) of the benchmark data set;

[0073] Fig. 22 is the ROC curve of the benchmark data set (Wilt); DETAILED DESCRIPTION

[0074] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] An unsupervised anomaly detection method based on local area angle information comprises the following steps:

[0076] S1: Receive a dataset D of outliers to be detected, where D is expressed as:

[0077] ;

[0078] in, represents the pth instance in D;

[0079] n represents the total number of instances in D;

[0080] d represents the dimension of the instance in D;

[0081] S2: Calculate the anomaly factor of each instance in D, where the anomaly factor of the pth instance in D is calculated as:

[0082] S2.1: Calculation of Nearest Neighbor Set ,in, Indicates distance Recent A collection of instances, is a preset parameter. In this embodiment, the distance is expressed as Euclidean distance;

[0083] S2.2: Calculation Inverse Nearest Neighbor Set , the calculation method is:

[0084] ;

[0085] in, is the qth instance in D, ;

[0086] S2.3: Calculation The impact set , the calculation method is:

[0087] ;

[0088] S2.4: Calculation The abnormal factor , the calculation method is:

[0089] ;

[0090] in, express The average of the midpoints is calculated as:

[0091] ;

[0092] S3: The larger the value, the The higher the probability of being an outlier, the higher the probability of being an outlier. According to the numerical value of the outlier factor of each instance in D, the instance identified as an outlier is given;

[0093] In this embodiment, the specific method for identifying an instance as abnormal is: according to the numerical value of the abnormal factor, all instances in D are arranged in descending order, and the first r instances are taken as abnormal instances, where r is a preset abnormal value number parameter.

[0094] Principle description:

[0095] like Figure 1 As shown, when The value of is 3, and the specific instance in D is , the circle with the instance as the center represents the k nearest neighbor set of the instance. The nearest neighbor set is: , , , , , .

[0096] The corresponding inverse The nearest neighbor set is: , , , , , .

[0097] Apparently Nearest Neighbor Set and Inverse The nearest neighbor set is asymmetric. For example, exist In, but not in middle; exist In, but In the present invention, we use the union of the k nearest neighbor set and the inverse k nearest neighbor set as the local area, that is, the influence set.

[0098] like Figures 2 to 4 As shown in the figure, each "x" represents an instance in the data set D. There are 2 clusters in D. The influence set is intuitively classified into three cases. There are three cases in total, namely Within the cluster, At the cluster boundaries, Outside the cluster, these three cases correspond to Figure 2 , 3 , 4;

[0099] (a) Inside a cluster: Example quilt The points in all directions are densely distributed. At this time, the points in the influence set are relatively uniform, and the average value is represents the center of these points.

[0100] (b) Cluster boundaries: Example At the edge of the cluster, The point distribution in Figure 2 That close means It may be part of a cluster or it may be an outlier.

[0101] (c) Outliers: Examples Farther away from the main cluster, Most of the points in The directions are closely distributed, indicating It is an outlier.

[0102] pass Figures 2 to 4From the toy dataset in , it can be found that the method proposed in the present invention can be applied to datasets containing multiple clusters.

[0103] In the influence set proposed in this invention, the instance distribution is related to the local area angle variance, which is obtained by measuring the vector and different vectors (in ). Figures 5 to 7 As shown in the figure, Represents one of the local area angles, that is, the vector and from To the impact set The angle between the vectors of an instance in .

[0104] It can be found that when When within a cluster, the local area angles are widely distributed in all directions; when When is an outlier, the local angles are concentrated in a smaller direction range. Therefore, the variance of these local angles can be used to reflect Is it an abnormal point? Abnormality factor formula.

[0105] Effect description:

[0106] We first evaluated the proposed unsupervised anomaly detection method on multiple synthetic datasets. Secondly, we used several benchmark datasets to validate the proposed method. The experimental results are reported by: Receiver Operating Characteristic curve (ROC curve), Area Under the ROC curve (AUC value), Precision at n (P@n) in the first n results, and Average Precision (AP).

[0107] To illustrate the effectiveness of the present invention, we first compare the local area angle information based anomaly detection method (LABOD) with angle-based anomaly detection (ABOD) and fast ABOD (FastABOD) to show the advantages of LABOD over the original model. Then, we compare LABOD with several classic or state-of-the-art unsupervised anomaly detection methods, including the local outlier factor method (LOF), the Antihubs method, the anomaly detection method based on angle intrinsic dimension (ABID), and the anomaly detection method based on boundary-aware local density (BLDOD). ABOD is an angle-based unsupervised anomaly detection algorithm, and FastABOD is a simplified version of ABOD. LOF is a classic density-based unsupervised anomaly detection algorithm. Antihubs is a high-dimensional unsupervised anomaly detection algorithm. ABID is another angle-based algorithm that has been successfully applied in intrinsic dimension estimation. BLDOD is a state-of-the-art unsupervised anomaly detection model. All codes are implemented in Matlab environment through mex interface and compiled using VC++2010. All experiments were run on a Windows 7 computer equipped with an Intel® Core™ i5-6200U CPU @2.30GHz and 16GB DRAM.

[0108] 1. Experiments based on artificial synthetic datasets

[0109] We first generated a Gaussian distribution dataset, a double Gaussian distribution with overlapping areas, and a double Gaussian distribution without overlapping areas. In the Gaussian distribution dataset, we generated 400 normal instances and 20 abnormal points. In the two double Gaussian distribution datasets with overlapping areas and without overlapping areas, we generated 800 normal instances and 40 abnormal points. When the dimension is set to 2, the artificial dataset is as follows Figures 8 to 10 As shown in the figure, the asterisk represents an ordinary instance, and the plus sign represents an outlier.

[0110] In FastABOD, LB-ABOD and LABOD algorithms, k was selected in the range of 20 to 200 with a step size of 20. The experiments were repeated 10 times. The best AUC, P@n and AP values ​​are reported in Table 1. The results are expressed as mean ± standard deviation (mean. ± std.).

[0111] Table 1. Results on artificial synthetic datasets

[0112]

[0113] The results in Table 1 show that LABOD outperforms ABOD and FastABOD. In particular, it can be found that ABOD performs poorly in the case of two Gaussian distributions that do not have overlapping regions. The reason is that ABOD is not suitable for multimodal datasets because it only considers local information. LABOD outperforms FastABOD because LABOD uses both the nearest neighbor set and the reverse nearest neighbor set.

[0114] Figures 11 to 13 The experimental results of double Gaussian distribution and Gaussian distribution are shown in Figure 2. Asterisks represent ordinary instances, plus signs represent outliers, and circles represent top-ranked samples. Figures 11 to 13 The top 40 samples are marked in Figures 11 to 13 As can be seen in the figure, the top instances in ABOD are all located in the upper right cluster (e.g. Fig.11 ), indicating that ABOD performs poorly in the double Gaussian distribution, while FastABOD and LABOD detect more outliers than ABOD in the double Gaussian distribution without overlapping areas.

[0115] In addition, to evaluate the impact of dimension on performance, we evaluated the LABOD algorithm on a high-dimensional double Gaussian distribution with no overlapping regions. The dimensions were set to 5, 10, 20, 50, and 100, respectively, and the experiment was repeated 10 times. The relevant results are shown in Figures 14 to 16 shown.

[0116] according to Figures 14 to 16 From the results, we can find that ABOD, FastABOD and LABOD are not sensitive to the dimension of artificial datasets; however, ABOD still performs poorly on high-dimensional double Gaussian distribution datasets.

[0117] 2. Experiments based on benchmark datasets

[0118] We use several benchmark datasets from (Radovanović et al., 2010, July) to validate the proposed method. The details of the benchmark datasets are listed in Table 2.

[0119] Table 2. Details of the benchmark datasets

[0120]

[0121] The number of features ranges from 3 to 259, and the number of instances ranges from 148 to 4839. In this experiment, we first compare LABOD with ABOD and FastABOD and present ROC curves, AUC values, and running times. In FastABOD and LABOD, the parameters The selection range is {20, 40, 60, 80, 100} and the AUC values ​​are listed in Table 3.

[0122] Table 3. Performance comparison of ABOD, FastABOD and LABOD on benchmark datasets

[0123]

[0124] As can be seen from Table 3, LABOD outperforms ABOD on the Lymphography, Ionosphere, and Wilt datasets, and outperforms FastABOD on all datasets. In the Wilt dataset, ABOD does not identify any outliers. The corresponding running times are shown in Table 4.

[0125] Table 4. Running time of ABOD, FastABOD, and LABOD on benchmark datasets

[0126]

[0127] As can be seen from Table 4, LABOD runs much faster than ABOD and FastABOD. Figure 6 Since ABOD failed to identify any outliers in the Wilt dataset, we only plotted the ROC curves of FastABOD and LABOD in the Wilt dataset.

[0128] exist Figures 17 to 22 In this paper, we draw the same conclusion as in Table 3. In addition, LABOD is also compared with LOF, AntiHub, ABID and BLDOD, and the results of AUC, P@n and AP are reported in Tables 5~7, respectively.

[0129] Table 5. Comparison of AUC values ​​of LOF, AntiHub, ABID, BLDOD, and LABOD on benchmark datasets

[0130]

[0131] From the results in Tables 5 to 7, we can see that LABOD outperforms LOF, AntiHub, ABID, and BLDOD on most benchmark datasets. Among them, LOF has the worst average result because LOF is sensitive to high-dimensional datasets.

[0132] Table 6. P@n comparison of LOF, AntiHub, ABID, BLDOD, and LABOD on benchmark datasets

[0133]

[0134] Table 7. Comparison of average precision (AP) of LOF, AntiHub, ABID, BLDOD and LABOD on benchmark datasets

[0135]

[0136] This paper proposes a local area angle information based anomaly detection (LABOD) method to solve the problems of previous angle-based anomaly detection (ABOD) methods: the internal structure of the data set is not considered and the time complexity is high. In LABOD, the score of each instance is determined by calculating the angular variance between the different vectors from the instance to the mean of its influence set, and the angular variance between the difference vectors from the instance to the instances in its influence set. Compared with FastABOD, an approximation method of ABOD, LABOD uses an influence set instead of a nearest neighbor set, and the amount of angle calculation is reduced from O( ²) is reduced to O( ). Experiments on synthetic datasets and multiple benchmark datasets show that LABOD outperforms ABOD and FastABOD in both performance and speed.

[0137] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An unsupervised anomaly detection method based on local area angle information, characterized in that: The following steps are involved: S1: Receive a dataset D of outliers to be detected, where D is expressed as: ； in, represents the pth instance in D; n represents the total number of instances in D; d represents the dimension of the instance in D; S2: Calculate the anomaly factor of each instance in D, where the anomaly factor of the pth instance in D is calculated as: S2.1: Calculation of Nearest Neighbor Set ,in, Indicates distance Recent A collection of instances, is the preset parameter; S2.2: Calculation Inverse Nearest Neighbor Set , the calculation method is: ; in, is the qth instance in D, ; S2.3: Calculation The impact set , the calculation method is: ； S2.4: Calculation The abnormal factor , the calculation method is: ； in, express The average of the midpoints is calculated as: ； S3: The larger the value, the The higher the probability of being an outlier, the higher the probability of being an outlier. According to the numerical value of the outlier factor of each instance in D, the instance identified as an outlier is given.

2. The unsupervised anomaly detection method based on local area angle information according to claim 1, characterized in that: The specific method for identifying an instance as an anomaly is as follows: according to the numerical value of the anomaly factor, all instances in D are sorted in descending order, and the first r instances are taken as anomaly instances, where r is the preset number of anomaly value parameters.

3. According to the unsupervised anomaly detection method based on local area angle information according to claim 1, it is characterized in that: calculate When , the distance is expressed as Euclidean distance.

4. An unsupervised anomaly detection device based on local area angle information, characterized in that: Includes the following modules: Data receiving module: used to receive the data set D of the abnormal points to be detected, where D is expressed as: ； in, represents the pth instance in D; n represents the total number of instances in D; d represents the dimension of the instance in D; Anomaly calculation module: used to calculate the anomaly factor of each instance in D, where the calculation method of the anomaly factor of the pth instance in D is: S2.1: Calculation of Nearest Neighbor Set ,in, Indicates distance Recent A collection of instances, is the preset parameter; S2.2: Calculation Inverse Nearest Neighbor Set , the calculation method is: ; in, is the qth instance in D, ; S2.3: Calculation The impact set , the calculation method is: ； S2.4: Calculation The abnormal factor , the calculation method is: ； in, express The average of the midpoints is calculated as: ； Output module: used to give the instances identified as abnormal according to the numerical value of the abnormal factor of each instance in D; The larger the value, the The higher the probability of being an outlier.

5. The unsupervised anomaly detection device based on local area angle information according to claim 4, characterized in that: The specific method for identifying an instance as an anomaly is as follows: according to the numerical value of the anomaly factor, all instances in D are sorted in descending order, and the first r instances are taken as anomaly instances, where r is the preset number of anomaly value parameters.

6. According to the unsupervised anomaly detection device based on local area angle information according to claim 4, it is characterized in that: calculate When , the distance is expressed as Euclidean distance.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 3 are implemented.

Images