A clustering-based outlier detection method and apparatus

Abstract

This invention provides a clustering-based outlier detection method and apparatus, comprising: performing cluster analysis on a historical dataset to form several effective clusters; removing outliers that cannot be integrated into any effective cluster to obtain a set of normal data points; constructing a tree index structure based on the set of normal data points using a tree algorithm; acquiring real-time data; performing a nearest neighbor search on the real-time data points within the set of normal data points using the tree index structure to locate the closest normal data point and the cluster center of the cluster to which the normal data point belongs; calculating the cluster center divergence factor of the real-time data point and comparing it with a preset threshold: if the cluster center divergence factor is less than or equal to the preset threshold, the real-time data point is determined to belong to the corresponding cluster and is a normal data point; if the cluster center divergence factor is greater than the preset threshold, the real-time data point is determined to deviate from the corresponding cluster and is an outlier. This method has the advantage of efficiently detecting abnormal data from large amounts of real-time data.

Description

Technical Field

[0001] This invention relates to the field of outlier detection technology, and more specifically, to a clustering-based outlier detection method and apparatus. Background Technology

[0002] Anomaly detection, also known as outlier detection or outlier point detection, is a method designed to identify and monitor anomalous behaviors or events in data. Anomalous behaviors or events typically refer to data points or samples that do not conform to general patterns or typical rules. Outlier detection is widely used in fields such as financial fraud detection, network intrusion detection, equipment failure detection, and medical diagnosis. Outlier detection primarily involves binary attributes, meaning it can only qualitatively determine whether something is normal or anomalous, rather than quantitatively indicating the degree of anomalousness.

[0003] Chinese invention patent CN114661795A discloses a method and apparatus for interpreting outliers in an athlete dataset based on the FOLOF algorithm. The method includes detecting outliers in an athlete dataset using the FOLOF algorithm: first, the elbow rule is used to determine the optimal number of clusters in the athlete dataset; then, the athlete dataset is pruned using the objective function of FCM to obtain a candidate set of athlete outliers; finally, the weighted local outlier detection algorithm is used to calculate the outlier degree of each athlete outlier in the candidate set, resulting in a set of athlete outliers. The outlier degree of each athlete outlier in the set is then divided into different levels using the golden section method, and athlete outlier analysis is performed based on the division results.

[0004] However, while the above-mentioned technical solutions can identify outliers in the dataset and quantify the degree of outlier, in fields such as healthcare, finance, electronic information, and equipment fault detection where there is a large amount of real-time data, they require repeated clustering operations on the real-time influx of data to complete outlier detection, resulting in high overall time complexity. This makes it impossible to meet the needs of fields such as healthcare, finance, electronic information, and equipment fault detection for rapid determination of abnormal states in new data.

[0005] In order to solve the above problems, people have been seeking an ideal technological solution. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a clustering-based outlier detection method and apparatus that can efficiently detect abnormal data from large amounts of real-time data and analyze the degree of abnormality.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] In a first aspect, the present invention provides a clustering-based outlier detection method, comprising:

[0009] Cluster analysis is performed on the historical dataset to form several effective clusters. Outliers that cannot be incorporated into any of the effective clusters are removed to obtain a set of normal data points.

[0010] Based on the set of normal data points, a tree index structure is constructed using a tree algorithm;

[0011] Acquire real-time data, and use the tree index structure to perform a nearest neighbor search on the real-time data points in the set of normal data points to locate the normal data point that is closest to the real-time data point and the cluster center of the cluster to which the normal data point belongs;

[0012] Calculate the cluster center divergence factor of the real-time data point and compare it with a preset threshold: if the cluster center divergence factor is less than or equal to the preset threshold, the real-time data point is determined to belong to the corresponding cluster and is a normal data point; if the cluster center divergence factor is greater than the preset threshold, the real-time data point is determined to deviate from the corresponding cluster and is an outlier.

[0013] The calculation of the cluster center divergence factor of the real-time data points includes:

[0014] Calculate the average distance between all normal data points in the cluster to which the normal data point belongs and the cluster center, and record it as the cluster center cohesion value of the cluster.

[0015] The distance between the real-time data point and the cluster center is calculated, and the ratio of this distance to the cluster center cohesion value is the cluster center divergence factor of the real-time data point.

[0016] Cluster analysis is used to screen outliers in historical data, establishing a high-quality benchmark dataset for subsequent detection and reducing noise interference. A tree-structured index optimizes nearest neighbor search efficiency, addressing the time-consuming issue of nearest neighbor queries in high-dimensional data and improving the response speed of real-time data detection. Furthermore, the tree-structured index is built solely on normal data, avoiding interference from outlier data. The degree of anomaly in real-time data points is quantitatively assessed based on the cluster core divergence factor, where the cluster core cohesion value reflects the density characteristics of the clusters, making anomaly detection more closely aligned with data distribution patterns and reducing false positives and false negatives. Historical data clustering and index construction only need to be performed once. The real-time detection phase only requires three core operations: nearest neighbor search, cluster core divergence factor calculation, and threshold comparison, all completed through the index. The streamlined process with no redundant calculations makes it widely applicable to various real-time monitoring scenarios such as financial fraud detection and equipment failure early warning.

[0017] In a preferred technical solution, the step of performing cluster analysis on the historical dataset to form several effective clusters includes:

[0018] Step S21: For each data point P in the historical dataset i Calculate data point P i The distance to its k-th nearest data point is taken as the data point P. i k-nearest neighbor distance d k (P i );

[0019] Step S22: For all data points P i k-nearest neighbor distance d k (P i Sort the data in ascending order and calculate the radius coverage distance r using equation (1). α cd;

[0020] (1)

[0021] Where α is a preset value, representing all data points P i The nearest distance d to k k (P i The radius coverage distance r is less than α. α The proportion of CD;

[0022] Step S23: Based on data point P i k-nearest neighbor distance d k (P i ) and the coverage distance r of radius α α The relationship between cd is used to determine the data point P. i The nearest cluster seed, specifically including:

[0023] If data point P i k-nearest neighbor distance d k (P i The radius coverage distance r is less than α. α cd, then the data point P i The k nearest neighbors are all seeds of the nearest cluster;

[0024] If data point P i k-nearest neighbor distance d k (P i () Greater than α radius coverage distance r α cd, and at least one data point O exists in the set of its k nearest neighbors. k Satisfy d k (O k The radius coverage distance r is less than α. α cd, then the data point O k Seed of the nearest cluster;

[0025] If data point P i k-nearest neighbor distance d k (Pi () Greater than α radius coverage distance r α cd, and there is no data point O in the set of its k nearest neighbors. k Satisfy d k (O k The radius coverage distance r is less than α. α cd, then amplify data point P i The k-th nearest neighbor is found to the (k+m)-th nearest neighbor until at least one data point O exists. k+m Satisfy d k (O k+m The radius coverage distance r is less than α. α cd, then the data point O k+m Seed of the nearest cluster;

[0026] Step S24: Take the union of the k nearest neighbors of the nearest cluster seed as the data point P. i The nearest local cluster.

[0027] By using the α radius to cover the distance, combined with real-world scenarios where normal data constitutes the majority, the dense neighborhood characteristics of normal data are characterized, improving the rationality of clustering selection; through a three-level nearest cluster seed determination logic, different data point distribution scenarios are covered, ensuring the integrity of normal data clustering and the accuracy of outlier identification;

[0028] In a preferred embodiment, the removal of outliers that cannot be integrated into any valid cluster to obtain a set of normal data points includes:

[0029] Step S31: Calculate the mean coordinates of all data points within the nearest local cluster, and use it as the nearest local cluster center;

[0030] Step S32: Calculate the average distance between each data point within the nearest local cluster and the nearest local cluster center, and use it as the cluster center cohesion value;

[0031] Step S33: Obtain data point P i The distance to the nearest local cluster center is then calculated, and the ratio of this distance to the cluster center cohesion value is used as the data point P. i Cluster center outward dispersion factor;

[0032] Step S34: Transfer data point P i The cluster center divergence factor is compared with a preset threshold: if the cluster center divergence factor is less than or equal to the preset threshold, then the data point P... i If a data point belongs to the nearest local cluster, it is considered a normal data point; if the cluster center divergence factor is greater than a preset threshold, then data point P... i A point that deviates from its nearest local cluster is identified as an outlier.

[0033] Step S35: For all data points P in the historical dataset i After the judgment is completed, remove the data point P that is determined to be an outlier. i This yields a set of normal data points.

[0034] The cluster center outlier factor enables quantitative assessment of outlier degree, breaking through the limitations of traditional binary qualitative interpretation and making outlier identification more objective and refined. The clustering process effectively filters isolated noise points through neighborhood association and cluster center calculation, providing a high-purity normal data benchmark for subsequent real-time detection.

[0035] In a preferred embodiment, the tree algorithm is either the KD Tree algorithm or the Ball Tree algorithm.

[0036] The KD Tree algorithm is suitable for low to medium-high dimensional data, while the Ball Tree algorithm performs better in high-dimensional data scenarios. The choice can be made flexibly according to the characteristics of the data dimension. The tree index structure reduces the time complexity of the nearest neighbor search from O(n) of brute force search to O(logn), where n is the number of data points, which greatly improves the query efficiency in high-data-volume scenarios and ensures the feasibility of real-time detection.

[0037] In a preferred embodiment, the step of constructing a tree-shaped index structure based on the set of normal data points using a tree algorithm specifically includes:

[0038] Step S41: Obtain the normal data corresponding to all data points identified as normal in the historical dataset, and form a normal dataset;

[0039] Step S42: Using the KD Tree algorithm, calculate the variance or data range of each dimension of the normal dataset, select the dimension with the largest variance or the largest data range as the split dimension of the root node, select the median of all data points under this split dimension as the split value, and divide the normal dataset into a left subset and a right subset according to the split value.

[0040] Step S43: Repeat step S42 for the left subset and the right subset respectively, recursively constructing subtrees until any of the following termination conditions are triggered: the subset contains only one data point, creating a leaf node; the subset is empty, returning an empty node; the preset maximum depth is reached; the number of data points in the subset is less than a preset threshold.

[0041] The dimension with the largest variance or the largest data range is selected as the split dimension to ensure that the split subspace can distinguish the data distribution to the greatest extent and improve the search efficiency of the index structure. The median is used as the split value to make the data volume of the left and right subsets as balanced as possible, avoid the tree structure skew, and ensure the stability of subsequent nearest neighbor searches. The recursion termination condition is clearly defined and can be flexibly adjusted according to the data scale and detection accuracy requirements, taking into account both index building efficiency and search accuracy. The index is built based on normal data to avoid the interference of abnormal data on the index structure and further improve the accuracy of real-time data nearest neighbor search.

[0042] In a preferred technical solution, the method further includes an anomaly dimension location step: for real-time data points identified as outliers, calculate the absolute difference between the data point and the cluster center of the cluster to which the nearest normal data point belongs in each data dimension, sort the absolute differences in descending order, and the dimension at the top of the sort is the main anomaly dimension of the outlier.

[0043] Breaking through the limitations of traditional outlier detection, which can only determine whether something is abnormal, this method can locate outliers at the anomaly dimension, providing a clear direction for subsequent problem investigation and root cause analysis. The location logic based on data dimension difference sorting is simple and efficient, requiring no additional complex algorithms, thus improving the method's practicality while ensuring detection efficiency. It can be directly applied to multi-dimensional data scenarios, helping users quickly focus on the root cause of anomalies and reduce the cost of problem investigation.

[0044] In a preferred embodiment, the distance between the real-time data point and the cluster center, the k-nearest neighbor distance, and the data point P i The distance to the nearest local cluster center, and the average distance between a data point and the nearest local cluster center, are calculated using only one of the following methods: Euclidean distance, Manhattan distance, or Minkowski distance.

[0045] Euclidean distance, Manhattan distance, or Minkowski distance are all distance metrics that can be adapted to different data characteristics and scenario requirements.

[0046] Secondly, the present invention provides a clustering-based outlier detection device, comprising:

[0047] The historical data outlier detection module performs cluster analysis on the historical dataset to form several effective clusters, removes outliers that cannot be integrated into any of the effective clusters, and obtains a set of normal data points.

[0048] The tree index structure construction module constructs a tree index structure based on the normal data point set using a tree algorithm.

[0049] The real-time data outlier detection module acquires real-time data and uses the tree index structure to perform a nearest neighbor search on the real-time data points in the set of normal data points to locate the normal data point closest to the real-time data point and the cluster center of the cluster to which the normal data point belongs.

[0050] Calculate the cluster center divergence factor of the real-time data point and compare it with a preset threshold: if the cluster center divergence factor is less than or equal to the preset threshold, the real-time data point is determined to belong to the corresponding cluster and is a normal data point; if the cluster center divergence factor is greater than the preset threshold, the real-time data point is determined to deviate from the corresponding cluster and is an outlier.

[0051] The calculation of the cluster center divergence factor of the real-time data points includes:

[0052] Calculate the average distance between all normal data points in the cluster to which the normal data point belongs and the cluster center, and record it as the cluster center cohesion value of the cluster.

[0053] The distance between the real-time data point and the cluster center is calculated, and the ratio of this distance to the cluster center cohesion value is the cluster center divergence factor of the real-time data point.

[0054] This invention has outstanding substantive features and significant progress compared to the prior art, specifically:

[0055] (1) By characterizing the dense neighborhood features of normal data through the α radius coverage distance, and combining the three-level nearest cluster seed determination logic and the quantitative evaluation of the cluster center dispersion factor, the interference of abnormal data in historical data can be effectively filtered out, and a high-purity normal data benchmark can be established for real-time detection.

[0056] (2) By using KD Tree or Ball Tree algorithms to construct a tree index structure, and with the simplified process of real-time nearest neighbor query and cluster center dispersion factor threshold determination, it can quickly respond to the continuous influx of real-time data, realize the instant judgment of abnormal state and the accurate location of abnormal dimension. Attached Figure Description

[0057] Figure 1 This is an example diagram of the α-radius coverage distance of the present invention;

[0058] Figure 2 This is an example diagram of the nearest local cluster center of the present invention;

[0059] Figure 3 This is an example diagram of real-time data anomaly detection according to the present invention;

[0060] Figure 4 This is an example diagram of the real-time data anomaly detection results of the present invention;

[0061] Figure 5This is an example diagram of real-time data classification according to the present invention;

[0062] Figure 6 This is an example of the real-time data anomaly dimension of the present invention. Figure 1 ;

[0063] Figure 7 This is an example of the real-time data anomaly dimension of the present invention. Figure 2 . Detailed Implementation

[0064] The technical solution of the present invention will be further described in detail below through specific embodiments.

[0065] Example 1

[0066] This embodiment discloses a clustering-based outlier detection method, including:

[0067] Cluster analysis is performed on the historical dataset to form several effective clusters. Outliers that cannot be incorporated into any of the effective clusters are removed to obtain a set of normal data points.

[0068] Based on the set of normal data points, a tree index structure is constructed using a tree algorithm;

[0069] Acquire real-time data, and use the tree index structure to perform a nearest neighbor search on the real-time data points in the set of normal data points to locate the normal data point that is closest to the real-time data point and the cluster center of the cluster to which the normal data point belongs;

[0070] Calculate the cluster center divergence factor of the real-time data point and compare it with a preset threshold: if the cluster center divergence factor is less than or equal to the preset threshold, the real-time data point is determined to belong to the corresponding cluster and is a normal data point; if the cluster center divergence factor is greater than the preset threshold, the real-time data point is determined to deviate from the corresponding cluster and is an outlier.

[0071] The calculation of the cluster center divergence factor of the real-time data points includes:

[0072] Calculate the average distance between all normal data points in the cluster to which the normal data point belongs and the cluster center, and record it as the cluster center cohesion value of the cluster.

[0073] The distance between the real-time data point and the cluster center is calculated, and the ratio of this distance to the cluster center cohesion value is the cluster center divergence factor of the real-time data point.

[0074] Outliers are essentially samples that deviate significantly from the overall data distribution pattern, while normal data typically exhibits dense distribution and the ability to form effective clusters. This embodiment first extracts the distribution characteristics of normal data through historical data clustering to establish a detection benchmark. Then, it optimizes the tree-based index to address the efficiency issue of nearest neighbor search under massive datasets, adapting to real-time detection scenarios. Finally, by quantitatively comparing the cluster center divergence factor of real-time data points with a threshold, it quickly matches real-time data with the normal benchmark, enabling immediate determination of abnormal states.

[0075] This embodiment uses cluster analysis on historical datasets to filter out normal data points and outliers. Normal data points exhibit a clustered distribution in multidimensional space, with enough similar samples in their neighborhoods to form effective clusters; while outliers, being significantly isolated from other data points, cannot be integrated into any dense clusters, thus achieving preliminary separation of the two types of data. Clustering filtering removes outlier noise from historical data, resulting in a high-purity set of normal data. This set serves as a reference standard for subsequent real-time detection, avoiding interference from outlier data on the detection benchmark and ensuring the accuracy of real-time judgment.

[0076] For massive real-time data in fields such as finance and electronic information, brute-force search by traversing all data points and calculating distances is extremely inefficient, with a time complexity of O(n), where n is the number of normal data points. This embodiment uses a tree algorithm to construct an index. Based on the dimensional characteristics of normal data points, such as the dimension with the largest variance or the largest data range, the multidimensional data space is recursively divided into several subspaces, forming a hierarchical tree structure. Each node corresponds to a subspace and stores the normal data points within that subspace. The tree structure allows for querying the nearest neighbors of real-time data points without traversing all normal data points. Instead, it only requires filtering subspaces that may contain the target nearest neighbors layer by layer along the tree structure, excluding subspaces that are too far away. This reduces the search time complexity to O(logn), significantly improving the efficiency of nearest neighbor queries for real-time data.

[0077] Whether a real-time data point is abnormal is determined by comparing it to the cluster to which the most recent normal data belongs. The cluster center is the geometric core of the normal data cluster, and its location accurately reflects the overall distribution center of the cluster, serving as a key reference for assessing whether a data point conforms to normal patterns. The preset threshold is determined based on the distribution of the cluster center outlier of normal data points within the cluster. For example, the upper limit of the statistical confidence interval of the cluster center outlier of normal data points represents the distribution boundary of normal data. If the cluster center outlier of a real-time data point does not exceed this threshold, it is considered to fall within the distribution range of normal data and is thus judged as normal. If it exceeds the threshold, it indicates that it deviates from the distribution boundary of the normal cluster and belongs to an outlier that does not conform to the overall pattern.

[0078] Example 2

[0079] like Figure 1-7As shown, this embodiment discloses a specific implementation of a clustering-based outlier detection method, including the following steps:

[0080] Step 1: Historical Dataset Preprocessing and Parameter Initialization

[0081] Standardize the data in the historical dataset to obtain n data points, each containing m-dimensional features (m≥1).

[0082] like Figure 1 As shown, the number of nearest neighbors k is determined by analyzing the stability of the average k-nearest neighbor distance for different k values. k can be adjusted according to the data density. Based on the general assumption that normal data constitutes the majority of the total, α is set to a value between 0.5 and 1. Setting α to 0.9 indicates that the k-nearest neighbor distance of 90% of normal data points is less than the α radius coverage distance. The k-nearest neighbor distances of all data points are sorted in ascending order, and the α radius coverage distance r is calculated based on α=0.9. α cd.

[0083] Step 2: Historical Data Cluster Analysis and Normal Data Screening

[0084] For each data point P in the historical dataset i The distance between a data point and its k nearest neighbors is calculated based on Euclidean distance, thus obtaining the k-nearest neighbor distance d for each data point. k (P i ).

[0085] Recently determined cluster seed:

[0086] If data point P i k-nearest neighbor distance d k (P i The radius coverage distance r is less than α. α cd, then the data point P i The k nearest neighbors are all the seeds of its nearest cluster;

[0087] If data point P i k-nearest neighbor distance d k (P i () Greater than α radius coverage distance r α cd, and at least one data point O exists in the set of its k nearest neighbors. k Satisfy d k (O k The radius coverage distance r is less than α. α cd, then the data point O k For data point P i Nearest cluster seed;

[0088] If data point P i k-nearest neighbor distance dk (P i () Greater than α radius coverage distance r α cd, and there is no data point O in the set of its k nearest neighbors. k Satisfy d k (O k The radius coverage distance r is less than α. α cd, then amplify data point P i The k-th nearest neighbor is found to the (k+m)-th nearest neighbor until at least one data point O exists. k+m Satisfy d k (O k+m The radius coverage distance r is less than α. α cd, then the data point O k+m For data point P i Recently clustered seed.

[0089] Taking data point P respectively i The union of the k nearest neighbors of the nearest cluster seed is used as P. i The nearest local cluster nlDens(P) i Each data point P i The set of nearest local clusters is nlDens(P); the process of solving for the nearest local clusters is shown below.

[0090]

[0091] In recent local clustering algorithms, the time complexity for querying a point in a KD-Tree is O(log n). , where m is the dimension of the data point.

[0092] like Figure 2 As shown, calculate nlDens(P) i The mean coordinates of all data points within a region are used to obtain the nearest local cluster center nlCenter(P). i ).

[0093] Calculate nlDens(P) i All points within the cluster are connected to the nearest local cluster center nlCenter(P). i The average distance is used as the cluster cohesion value.

[0094] Calculate P i To nlCenter(P i The distance between the cluster centers is calculated, and the ratio of this distance to the cluster center cohesion value is used to obtain the cluster center divergence factor DensCDF.

[0095] The preset threshold offset is the α percentile of the DensCDF values ​​of all data points. Data points with DensCDF > offset are identified as outliers, and data points with DensCDF ≤ offset are identified as normal data points. Data points P identified as outliers are removed. i The purified normal dataset is then constructed. The process of solving the cluster center dispersion factor of the data points is shown below.

[0096]

[0097] The cluster center divergence factor FS of n data points is obtained, denoted by PNArr, which indicates whether the n data points are normal or abnormal. The transformation process is as follows:

[0098] PNArr = zeros(n);

[0099] offset = percentile(FS, 100 * r) α cd);

[0100] PNArr[FS <offset]=1;

[0101] After the transformation, each point represented by PNArr has a binary attribute, that is, 0 represents abnormality and 1 represents normality.

[0102] Step 3: Building the tree index structure

[0103] Based on a normal dataset, a tree-like index structure is constructed using the KD Tree algorithm:

[0104] Calculate the variance of each dimension of the normal dataset, select the dimension with the largest variance as the splitting dimension of the root node, select the median of all data points under this dimension as the splitting value, and divide the dataset into a left subset and a right subset;

[0105] Repeat the above segmentation steps for the left and right subsets respectively, recursively constructing subtrees until any termination condition is triggered: the subset contains only one data point (create a leaf node), the subset is empty (return an empty node), the preset maximum depth is reached, or the number of data points in the subset is less than a preset threshold, thus completing the construction of the tree index.

[0106] Step 4: Real-time outlier detection

[0107] Acquire real-time data points and perform missing value imputation and standardization according to the preprocessing standards of historical datasets to ensure data format consistency;

[0108] like Figure 3As shown, a nearest neighbor search is performed on the real-time data points using the constructed tree index to quickly locate the normal data points closest to the real-time data points and the cluster centers of the clusters to which the normal data points belong; based on all data points P in the normal dataset... i The DensCDF distribution is used to establish a confidence interval as a preset threshold. The cluster center divergence factor of real-time data points is calculated. If the cluster center divergence factor of a real-time data point does not exceed the preset threshold, it is determined to be a normal data point; if it exceeds the preset threshold, it is determined to be an outlier. The process of solving the cluster center divergence factor of real-time data points is as follows.

[0109]

[0110] like Figure 4 As shown, normal data is extracted from anomalous data, and a KD Tree is constructed based on this to perform real-time data detection. In (a), the size of the red outer circle of each data point represents the size of the outlier value given by DensCDF. (b) and (c) show that small-radius red circles are retained, while large-radius red circles represent outliers and are filtered out. (d) Based on the retained normal data points, it can be seen by establishing equipotential lines in the data field that the farther away from the cluster of normal data points, the greater the probability of an anomaly.

[0111] like Figure 5 As shown, the left figure represents two data point clusters, and the right figure represents the DensCDF value of the real-time data relative to the two data point clusters. Based on the magnitude of the DensCDF value relative to the two clusters, it is determined which cluster the data belongs to, thereby completing the real-time data classification task.

[0112] Step 5: Anomaly Dimension Localization

[0113] like Figure 6-7 As shown, if a real-time data point is identified as an outlier, the difference between the data point and the nearest local cluster center in each dimension is calculated. The differences are sorted in descending order, and the dimension with the largest difference is the main abnormal dimension of the outlier, thus achieving accurate location of the root cause of the anomaly.

[0114] Example 3

[0115] This embodiment uses fetal cardiac monitoring data (Cardiotocography dataset) as the application scenario and discloses the specific implementation process of a clustering-based outlier detection method.

[0116] The Cardiotocography dataset contains 1655 normal samples, 295 suspicious samples, and 176 pathological samples. The method in this embodiment identifies abnormal data in pathological and suspicious samples, enabling early warning of fetal health status.

[0117] Step 1: Historical data preprocessing and parameter initialization

[0118] The Cardiotocography dataset was analyzed, and 21 numerical features related to fetal heart rate and uterine contractions were extracted to form a feature matrix X. Since there were no missing values ​​in the dataset, standardization was performed directly, and the RobustScaler method was used to reduce the interference of extreme values ​​on the detection results and ensure the stability of the data distribution.

[0119] By analyzing the stability of the average k-nearest neighbor distance for different k values, k=30 was determined, meaning each data point has 30 nearest neighbors. Based on the assumption that normal data constitutes the majority, α=0.95 was set, meaning that the k-nearest neighbor distance of 95% of normal data points is less than the α radius coverage distance. The k-nearest neighbor distances of all data points were sorted in ascending order, and the α radius coverage distance r was calculated based on α=0.95. α cd=2.35.

[0120] Step 2: Historical Data Cluster Analysis and Normal Data Screening

[0121] For each data point P in the dataset i The k-nearest neighbor distance d of each data point is obtained by calculating its distance to the 30 nearest neighbors based on Euclidean distance. k (P i ).

[0122] (1) Determining the nearest cluster seed:

[0123] If data point d k (P i If ) < 2.35, then the 30 nearest neighbors of this data point are all data point P. i The nearest cluster seed is the absolute k-nearest neighbor;

[0124] If data point d k (P i ) > 2.35, but among its 30 nearest neighbors, there exists d k (O k Data point O < 2.35 k , will O k As data point P i The nearest cluster seed is a non-absolute k-nearest neighbor;

[0125] If data point d k (P i ) > 2.35, and its 30 nearest neighbors d k (O k If all values ​​are greater than 2.35, expand the nearest neighbor search range to the 30+mth point until d is found. k (Ok+m For data points where ) < 2.35, O k+m As data point P i Recently clustered seeds.

[0126] (2) Calculation of nearest local clusters and cluster centers:

[0127] With data point P i The union of the 30 nearest neighbors of the nearest cluster seed is taken as the nearest local cluster. The mean coordinates of all data points in the nearest local cluster are calculated based on the Euclidean distance to obtain the nearest local cluster center. Then, the average distance from all points in the cluster to the cluster center is calculated as the cluster center cohesion value.

[0128] (3) Calculation of cluster center divergence factor and data classification:

[0129] Calculate data point P i The ratio of the distance to the nearest local cluster center to the cluster center cohesion value, i.e., the value of data point P. i DensCDF (P) cluster center outward dispersion factor i );

[0130] Statistical analysis of all data points P i The DensCDF value distribution was analyzed, and a threshold offset was set at the 95th percentile of the DensCDF value. Data points with DensCDF > offset were identified as outliers, accounting for 5% of the total, corresponding to pathological samples and some suspicious samples; data points with DensCDF ≤ offset were identified as normal data points; after removing outliers, a cleaned-up normal dataset was formed.

[0131] Step 3: Building the tree index structure

[0132] Based on the filtered normal dataset, a tree index structure is constructed using the KD Tree algorithm:

[0133] Calculate the variance of the 21 feature dimensions of the normal dataset, select the dimension with the largest variance as the splitting dimension of the root node, select the median of all data points under this dimension as the splitting value, and divide the dataset into a left subset and a right subset.

[0134] Repeat the above segmentation steps for the left and right subsets, recursively constructing subtrees until the subset contains only one data point or reaches the preset maximum depth, thus completing the KD Tree index construction.

[0135] Step 4: Real-time data anomaly detection

[0136] Real-time fetal heart monitoring data was acquired, and the data was standardized using RobustScaler in the same way as historical data. 21 features were extracted to form real-time data points.

[0137] The constructed KD Tree index is used to perform a nearest neighbor search on the real-time data points to find the closest normal data points and the cluster center of the cluster to which the normal data points belong. Then, based on the cluster center, the cluster center divergence factor of the real-time data points is calculated.

[0138] Based on the DensCDF distribution of the normal dataset, a confidence interval covering 99.7% of normal points was established as a preset threshold. If the cluster center outlier of the real-time data points does not exceed this threshold, it is judged as normal data and the fetus is in good health. If it exceeds the threshold, it is judged as an outlier, which poses a health risk and requires further intervention.

[0139] In this embodiment, with k=30, the accuracy is superior to the LOF and iForest algorithms, with accuracy, recall and precision all being 0.95; it can effectively reduce false alarms caused by individual differences and achieve early warning of fetal abnormalities.

[0140] Example 4

[0141] This embodiment uses the time series data of aircraft flywheel rotation speed as the application scenario and discloses a specific implementation method for outlier detection based on clustering.

[0142] This embodiment combines multi-domain feature extraction with the method of this invention to achieve real-time detection and location of abnormal flywheel speed, thereby ensuring the safe operation of the aircraft.

[0143] Step 1: Historical Data Preprocessing and Feature Extraction

[0144] Data on the normal rotational speed trend of aircraft flywheels was collected in 500 cases, and data on abnormal fluctuation trends were collected in 2380 cases. The sliding window size w was set, and the flywheel rotational speed sequence within each window was x=[x1, x2, ..., x...]. w ];N w This indicates the number of data points within the window.

[0145] Based on the flywheel speed characteristics, 11 features were designed as follows:

[0146] (1) Basic statistical characteristics, reflecting the distribution characteristics of the rotation speed sequence within the sliding window.

[0147] Moving standard deviation: measures the degree of fluctuation in engine speed.

[0148] ;

[0149] Moving range: reflects the maximum range of speed variation.

[0150] ;

[0151] Moving average absolute deviation: characterizes the average degree to which the rotational speed deviates from the central trend.

[0152] .

[0153] (2) Rate of change characteristics, capturing the dynamic change characteristics of rotational speed.

[0154] First-order difference absolute value: ;

[0155] Second-order difference (acceleration): .

[0156] (3) Peak detection characteristics;

[0157] Local maxima ratio: Frequency of abnormal peak values ​​detected

[0158] ;

[0159] Where, N localmax This represents the number of local maxima within the window.

[0160] (4) Distribution characteristics;

[0161] Skewness: measures the asymmetry of rotational speed distribution.

[0162] ;

[0163] Kurtosis: Characterizes the thickness of the tail of the distribution.

[0164] .

[0165] (5) Autocorrelation characteristics

[0166] Lag autocorrelation coefficient: assesses the short-term memory of a sequence.

[0167] .

[0168] (6) Energy characteristics

[0169] Energy density: characterizing signal strength

[0170] .

[0171] (7) Zero crossover rate, which reflects the signal oscillation frequency.

[0172] ;

[0173] in, This is an indicator function.

[0174] (8) Frequency domain characteristics;

[0175] Dominant frequency: Extracting the main components in the frequency domain

[0176] ;

[0177] in, For periodic spectrum estimation.

[0178] (9) Entropy characteristics

[0179] Approximate entropy: Quantization sequence complexity

[0180] .

[0181] (10) Trend characteristics

[0182] Linear trend slope: capturing local trends of change

[0183] .

[0184] (11) Synthetic characteristics

[0185] Volatility Index: A comprehensive indicator of the degree of speed fluctuation.

[0186] ;

[0187] Regularity index: quantifying sequence self-similarity

[0188] .

[0189] Convert the flywheel speed into a characteristic. Where d is the feature dimension. Then, with w as the window size, the data from Xanomaly are sequentially merged with Xnormal to form the feature matrix. .

[0190] After data standardization, the high-dimensional feature matrix extracted is reduced using the KPCA linear kernel, retaining the first two principal components to reduce computational complexity.

[0191] Step 2: Historical Data Cluster Analysis and Normal Data Screening

[0192] k is determined through stability analysis, α is set, the k-nearest neighbor distances of all data points are calculated and sorted, and the coverage distance r of the α radius is obtained. α cd;

[0193] According to the method of Example 2 or 3, based on the k-nearest neighbor distance and r α The relationship between cd determines the nearest cluster seed for each data point, constructs the nearest local cluster, and calculates the nearest local cluster center and cluster center cohesion value;

[0194] Calculate the DensCDF value for each data point, set a threshold to filter out normal data points, that is, the feature data corresponding to the normal operation of the flywheel, and form a normal dataset.

[0195] Step 3: Building the tree index structure

[0196] Based on a normal dataset, an index is built using the Ball Tree algorithm. The dimension with the largest data range among the two principal component dimensions after dimensionality reduction is selected as the split dimension. The median of this dimension is used as the split value. The dataset is recursively divided to build subtrees until the termination condition is met, thus completing the index construction.

[0197] Step 4: Real-time data anomaly detection and anomaly dimension localization

[0198] Real-time data processing and detection: Real-time flywheel speed time-series data is acquired, and 11 types of features are extracted through a sliding window. After dimensionality reduction and standardization by KPCA, real-time data points are obtained. Nearest neighbor search is performed using a Ball Tree index to locate the normal data point closest to the real-time data point and the cluster center of the cluster to which the normal data point belongs. Based on the DensCDF distribution of the normal dataset, a preset threshold is determined, the cluster center divergence factor of the real-time data point is calculated, and compared with the preset threshold to determine whether the real-time data point is a normal or an outlier.

[0199] If a real-time data point is identified as an outlier, the difference between the real-time data point and the nearest local cluster center in 11 feature dimensions is calculated. The differences are sorted in descending order, and the dimension with the largest difference is the main abnormal dimension. For example, an abnormal peak detection feature dimension indicates that the flywheel has sudden peak fluctuations, and an abnormal frequency domain feature dimension indicates that the rotational speed frequency distribution is abnormal. By locating the abnormal dimension, the root cause of the anomaly can be quickly identified, providing a clear direction for troubleshooting.

[0200] Example 5

[0201] This embodiment discloses a clustering-based outlier detection device, comprising:

[0202] The historical data outlier detection module performs cluster analysis on the historical dataset to form several effective clusters, removes outliers that cannot be integrated into any of the effective clusters, and obtains a set of normal data points.

[0203] The tree index structure construction module constructs a tree index structure based on the normal data point set using a tree algorithm.

[0204] The real-time data outlier detection module acquires real-time data and uses the tree index structure to perform a nearest neighbor search on the real-time data points in the set of normal data points to locate the normal data point closest to the real-time data point and the cluster center of the cluster to which the normal data point belongs.

[0205] Calculate the cluster center divergence factor of the real-time data point and compare it with a preset threshold: if the cluster center divergence factor is less than or equal to the preset threshold, the real-time data point is determined to belong to the corresponding cluster and is a normal data point; if the cluster center divergence factor is greater than the preset threshold, the real-time data point is determined to deviate from the corresponding cluster and is an outlier.

[0206] The calculation of the cluster center divergence factor of the real-time data points includes:

[0207] Calculate the average distance between all normal data points in the cluster to which the normal data point belongs and the cluster center, and record it as the cluster center cohesion value of the cluster.

[0208] The distance between the real-time data point and the cluster center is calculated, and the ratio of this distance to the cluster center cohesion value is the cluster center divergence factor of the real-time data point.

[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A clustering-based outlier detection method, characterized in that, include: Cluster analysis is performed on the historical dataset to form several effective clusters. Outliers that cannot be incorporated into any of the effective clusters are removed to obtain a set of normal data points. Based on the set of normal data points, a tree index structure is constructed using a tree algorithm; Acquire real-time data, and use the tree index structure to perform a nearest neighbor search on the real-time data points in the set of normal data points to locate the normal data point that is closest to the real-time data point and the cluster center of the cluster to which the normal data point belongs; Calculate the cluster center divergence factor of the real-time data point and compare it with a preset threshold: if the cluster center divergence factor is less than or equal to the preset threshold, then the real-time data point is determined to belong to the corresponding cluster and is a normal data point; If the cluster center divergence factor is greater than a preset threshold, the real-time data point is determined to deviate from the corresponding cluster and is an outlier. The calculation of the cluster center divergence factor of the real-time data points includes: Calculate the average distance between all normal data points in the cluster to which the normal data point belongs and the cluster center, and record it as the cluster center cohesion value of the cluster. The distance between the real-time data point and the cluster center is calculated, and the ratio of this distance to the cluster center cohesion value is the cluster center divergence factor of the real-time data point.

2. The clustering-based outlier detection method according to claim 1, characterized in that, The clustering analysis of the historical dataset forms several effective clusters, including: Step S21: For each data point P in the historical dataset i Calculate data point P i The distance to its k-th nearest data point is taken as the data point P. i k-nearest neighbor distance d k (P i ); Step S22: For all data points P i k-nearest neighbor distance d k (P i Sort the data in ascending order and calculate the radius coverage distance r using equation (1). α cd; （1） Where α is a preset value, representing all data points P i The nearest distance d to k k (P i The radius of the coverage distance r is less than α. α The proportion of CD; Step S23: Based on data point P i k-nearest neighbor distance d k (P i ) and the coverage distance r of radius α α The relationship between cd is used to determine the data point P. i The nearest cluster seed, specifically including: If data point P i k-nearest neighbor distance d k (P i The radius of the coverage distance r is less than α. α cd, then the data point P i The k nearest neighbors are all seeds of the nearest cluster; If data point P i k-nearest neighbor distance d k (P i () Greater than α radius coverage distance r α cd, and at least one data point O exists in the set of its k nearest neighbors. k Satisfy d k (O k The radius of the coverage distance r is less than α. α cd, then the data point O k Seed of the nearest cluster; If data point P i k-nearest neighbor distance d k (P i () Greater than α radius coverage distance r α cd, and there is no data point O in the set of its k nearest neighbors. k Satisfy d k (O k The radius of the coverage distance r is less than α. α cd, then amplify data point P i The k-th nearest neighbor is found to the (k+m)-th nearest neighbor until at least one data point O exists. k+m Satisfy d k (O k+m The radius of the coverage distance r is less than α. α cd, then the data point O k+m Seed of the nearest cluster; Step S24: Take the union of the k nearest neighbors of the nearest cluster seed as the data point P. i The nearest local cluster.

3. The clustering-based outlier detection method according to claim 2, characterized in that, The removal of outliers that cannot be integrated into any valid cluster, resulting in a set of normal data points, includes: Step S31: Calculate the mean coordinates of all data points within the nearest local cluster, and use it as the nearest local cluster center; Step S32: Calculate the average distance between each data point within the nearest local cluster and the nearest local cluster center, and use it as the cluster center cohesion value; Step S33: Obtain data point P i The distance to the nearest local cluster center is then calculated, and the ratio of this distance to the cluster center cohesion value is used as the data point P. i Cluster center outward dispersion factor; Step S34: Transfer data point P i The cluster center divergence factor is compared with a preset threshold: if the cluster center divergence factor is less than or equal to the preset threshold, then the data point P... i If a data point belongs to the nearest local cluster, it is considered a normal data point; if the cluster center divergence factor is greater than a preset threshold, then data point P... i A point that deviates from its nearest local cluster is identified as an outlier. Step S35: For all data points P in the historical dataset i After the judgment is completed, remove the data point P that is determined to be an outlier. i This yields a set of normal data points.

4. The clustering-based outlier detection method according to claim 3, characterized in that, The tree algorithm is either the KDTree algorithm or the Ball Tree algorithm.

5. The clustering-based outlier detection method according to claim 4, characterized in that, The construction of a tree-based index structure based on the set of normal data points using a tree algorithm specifically includes: Step S41: Obtain the normal data corresponding to all data points identified as normal in the historical dataset, and form a normal dataset; Step S42: Using the KD Tree algorithm, calculate the variance or data range of each dimension of the normal dataset, select the dimension with the largest variance or the largest data range as the split dimension of the root node, select the median of all data points under this split dimension as the split value, and divide the normal dataset into a left subset and a right subset according to the split value. Step S43: Repeat step S42 for the left subset and the right subset respectively, recursively constructing subtrees until any of the following termination conditions are triggered: the subset contains only one data point, creating a leaf node; the subset is empty, returning an empty node; the preset maximum depth is reached; the number of data points in the subset is less than a preset threshold.

6. The clustering-based outlier detection method according to claim 5, characterized in that, It also includes an anomaly dimension location step: for real-time data points identified as outliers, calculate the absolute difference between the data point and the cluster center of the cluster to which the closest normal data point belongs in each data dimension, sort the absolute differences in descending order, and the dimension at the top of the sort is the main anomaly dimension of the outlier.

7. The clustering-based outlier detection method according to claim 6, characterized in that, The distance between the real-time data point and the cluster center, the k-nearest neighbor distance, and the data point P i The distance to the nearest local cluster center, and the average distance between a data point and the nearest local cluster center, are calculated using only one of the following methods: Euclidean distance, Manhattan distance, or Minkowski distance.

8. A clustering-based outlier detection device, characterized in that, include: The historical data outlier detection module performs cluster analysis on the historical dataset to form several effective clusters, removes outliers that cannot be integrated into any of the effective clusters, and obtains a set of normal data points. The tree index structure construction module constructs a tree index structure based on the normal data point set using a tree algorithm. The real-time data outlier detection module acquires real-time data and uses the tree index structure to perform a nearest neighbor search on the real-time data points in the set of normal data points to locate the normal data point closest to the real-time data point and the cluster center of the cluster to which the normal data point belongs. Calculate the cluster center divergence factor of the real-time data point and compare it with a preset threshold: if the cluster center divergence factor is less than or equal to the preset threshold, then the real-time data point is determined to belong to the corresponding cluster and is a normal data point; If the cluster center divergence factor is greater than a preset threshold, the real-time data point is determined to deviate from the corresponding cluster and is an outlier. The calculation of the cluster center divergence factor of the real-time data points includes: Calculate the average distance between all normal data points in the cluster to which the normal data point belongs and the cluster center, and record it as the cluster center cohesion value of the cluster. The distance between the real-time data point and the cluster center is calculated, and the ratio of this distance to the cluster center cohesion value is the cluster center divergence factor of the real-time data point.

Images