An outlier detection method and apparatus

Abstract

This invention discloses a method and apparatus for outlier detection, relating to the field of computer technology. One specific implementation of the method includes: dividing the dataset to be detected into a set of clusters and a set of suspected outliers; filtering suspected outliers in the set of suspected outliers based on the set of clusters; determining the weighted density anomaly degree of each suspected outlier in the filtered set of suspected outliers; and determining the outliers in the dataset to be detected based on the weighted density anomaly degree of each suspected outlier. This embodiment can reduce the time complexity of outlier detection, improve detection efficiency, and enhance detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for outlier detection. Background Technology

[0002] Currently, clustering-based outlier detection algorithms or schemes all involve first clustering, and then identifying outliers by the opposition between normal points and outliers in the cluster.

[0003] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:

[0004] Outlier detection has high time complexity, low efficiency, and poor accuracy. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an outlier detection method and apparatus, which can reduce the time complexity of outlier detection, improve detection efficiency, and enhance detection accuracy.

[0006] To achieve the above objectives, according to one aspect of the present invention, an outlier detection method is provided, comprising:

[0007] The dataset to be detected is divided into a set of clusters and a set of suspected outliers;

[0008] Filter suspected outliers in the suspected outlier set based on the cluster set;

[0009] Determine the degree of weighted density anomaly of each suspected outlier in the filtered set of suspected outliers, and determine the outliers in the dataset to be detected based on the degree of weighted density anomaly of each suspected outlier.

[0010] Optionally, the dataset to be detected can be divided into a set of clusters and a set of suspected outliers, including:

[0011] The dataset to be detected is split to obtain multiple clusters;

[0012] For any cluster among multiple clusters: if the difference between the number of sample points of any cluster and the preset cluster capacity does not exceed a preset threshold, add any cluster to the cluster set; if the difference between the number of sample points of any cluster and the preset cluster capacity exceeds a preset threshold, add any cluster to the suspected outlier set.

[0013] Optionally, after dividing the dataset to be detected into a set of clusters and a set of suspected outliers, the process may also include:

[0014] The clusters with unevenly distributed sample points in the suspected outlier set are split again to obtain two new clusters;

[0015] Compare the number of sample points in the two new clusters, and remove the sample points in the new cluster with more sample points from the suspected outlier set.

[0016] Optionally, filtering of suspected outliers in the suspected outlier set based on the cluster set includes:

[0017] For any suspected outlier in the set of suspected outliers, determine the cluster in the cluster set that is closest to or second closest to any suspected outlier;

[0018] Determine the difference in information entropy before and after adding any suspected outlier to the nearest or second nearest cluster;

[0019] If the information entropy difference does not exceed the preset change threshold, suspected outliers are removed from the suspected outlier set.

[0020] Optionally, the weighted density anomaly degree of each suspected outlier in the filtered set of suspected outliers is determined, including:

[0021] Determine the splitting round of each suspected outlier in the filtered set of suspected outliers, and weight the original outlier factor of the suspected outlier according to the splitting round to obtain the splitting round weighting factor of the suspected outlier.

[0022] The local density of suspected outliers is determined based on the splitting round weighting factor of suspected outliers;

[0023] The weighted density anomaly degree of suspected outliers is calculated based on the splitting round weighting factor and local density.

[0024] Optionally, the original outlier factor of the suspected outlier is obtained through the following steps:

[0025] If the maximum distance between a suspected outlier and a sample point in the nearest cluster is less than the average distance between a suspected outlier and a sample point in the second nearest cluster, then the number of neighbors of the suspected outlier is determined based on the nearest cluster.

[0026] If the maximum distance between the suspected outlier and the sample point in the nearest cluster is greater than or equal to the average distance between the suspected outlier and the sample point in the second nearest cluster, then the number of neighborhoods of the suspected outlier is determined based on the nearest cluster and the second nearest cluster.

[0027] The original outlier factor is determined based on the number of neighborhoods of suspected outliers.

[0028] Optionally, outliers in the dataset to be detected are determined based on the weighted density anomaly degree of each suspected outlier, including:

[0029] If the weighted density anomaly of a suspected outlier exceeds a preset outlier threshold, the suspected outlier is identified as an outlier in the dataset to be detected.

[0030] According to a second aspect of the present invention, an outlier detection device is provided, comprising:

[0031] The partitioning module is used to divide the dataset to be detected into a set of clusters and a set of suspected outliers;

[0032] The filtering module is used to filter suspected outliers in the suspected outlier set based on the cluster set;

[0033] The determination module is used to determine the weighted density anomaly degree of each suspected outlier in the filtered suspected outlier set, and to determine the outliers in the dataset to be detected based on the weighted density anomaly degree of each suspected outlier.

[0034] According to a third aspect of the present invention, an electronic device is provided, comprising:

[0035] One or more processors;

[0036] Memory, used to store one or more programs.

[0037] When one or more programs are executed by one or more processors, the one or more processors implement the methods of any of the above embodiments.

[0038] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method of any of the above embodiments.

[0039] According to a fifth aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the method of any of the above embodiments.

[0040] One embodiment of the above invention has the following advantages or beneficial effects: by dividing the dataset to be detected into a set of clusters and a set of suspected outliers; filtering the suspected outliers in the set of suspected outliers based on the set of clusters; determining the weighted density anomaly degree of each suspected outlier in the filtered set of suspected outliers; and determining the outliers in the dataset to be detected based on the weighted density anomaly degree of each suspected outlier; thereby reducing the time complexity of outlier detection, improving detection efficiency, and increasing detection accuracy.

[0041] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0042] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0043] Figure 1 This is a schematic diagram of the main flow of the outlier detection method according to an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the main flow of an outlier detection method according to a preferred embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of global suspected outliers according to a preferred embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of a suspected outlier point between sparse clusters and dense clusters according to a preferred embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of a density-uniform cluster containing suspected outliers according to a preferred embodiment of the present invention;

[0048] Figure 6 This is the original image obtained by applying outlier detection according to a preferred embodiment of the present invention;

[0049] Figure 7 This is a schematic diagram illustrating the detection effect of outlier detection according to a preferred embodiment of the present invention;

[0050] Figure 8 This is a schematic diagram of the main modules of the outlier detection device according to an embodiment of the present invention;

[0051] Figure 9 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0052] Figure 10 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0053] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0054] It should be noted that the acquisition, storage, and application of personal information involved in the embodiments of the present invention comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0055] Currently, clustering-based outlier detection algorithms or schemes all involve first clustering, and then identifying outliers by the opposition between normal points and outliers in the cluster. This results in high time complexity, low detection efficiency, and poor detection accuracy for outlier detection.

[0056] In view of this, according to one aspect of the present invention, an outlier detection method is provided.

[0057] Figure 1 This is a schematic diagram of the main flow of the outlier detection method according to an embodiment of the present invention. Figure 1 As shown, the outlier detection method according to an embodiment of the present invention includes the following steps S101 to S103.

[0058] Step S101: Divide the dataset to be detected into a set of clusters and a set of suspected outliers.

[0059] The dataset to be analyzed refers to the dataset that needs to be processed. This dataset contains all the sample points to be analyzed; some of these points may belong to normal clusters, while others may be outliers. When processing the dataset, the first step is to perform preliminary analysis and partitioning, dividing it into cluster sets and suspected outlier sets. The cluster set refers to the set of sample points representing normal or dominant patterns in the dataset. The dataset is grouped into several highly similar clusters using a clustering algorithm, with each cluster containing sample points that are close to each other in a specific feature space. The suspected outlier set refers to the set of sample points initially identified as potentially deviating from normal patterns. These sample points may be located near the boundaries of clusters, or their feature values ​​may be significantly different from most sample points.

[0060] Specifically, clustering algorithms such as FUSHC (Fast and uniform segmentation hierarchical clustering) or DBSCAN (Density-Based Spatial Clustering of Applications with Noise) can be used to partition the data. FUSHC hierarchical clustering progressively merges or splits clusters by constructing a tree-like structure, thus forming a set of clusters. When identifying a set of suspected outliers, outlier detection algorithms, such as Local Outlier Factor (LOF) or Isolation Forest, can be applied. These algorithms identify points that are significantly different from other points by analyzing the density or anomalous behavior of the sample points, thus forming a set of suspected outliers. For example, the FUSHC algorithm can divide sample points into multiple clusters, each representing a set of sample points. Methods such as LOF or Isolation Forest can then be used to detect suspected outliers that are significantly different from the majority of sample points, thus dividing the dataset into a set of clusters and a set of suspected outliers.

[0061] Step S102: Filter the suspected outliers in the suspected outlier set based on the cluster set.

[0062] Information from the cluster set can reveal the distribution of normal sample points in the dataset. Each cluster represents a common pattern or group in the data, and the sample points within it have high similarity. Sample points in the suspected outlier set, although initially identified as potential anomalies, may be marked as suspected outliers because they are located at cluster boundaries or their feature values ​​deviate from the mainstream distribution.

[0063] During the filtering process, the relationship between suspected outliers and the cluster set can be analyzed. For example, each suspected outlier can be compared with all clusters in the cluster set, and its distance to the center of each cluster can be calculated. This distance information helps determine whether the point truly has a significant connection with any cluster. A suspected outlier that is far from the cluster center is likely an outlier. Conversely, if a suspected outlier is close to some cluster centers, it may be a misjudgment due to the complexity of the data distribution or the ambiguity of the cluster boundaries, and it should be removed from the suspected outlier set. Alternatively, suspected outliers can be added to the nearest clusters, and the impact of this addition on the overall cluster structure can be evaluated. If adding these points does not significantly change the cluster density or distribution characteristics, it indicates that these points may belong to the extended regions of these clusters rather than being true outliers. In this way, points that are more reasonable in the data distribution can be removed from the suspected outlier set, thereby reducing misjudgments. After filtering based on the cluster set, the remaining suspected outlier set will more accurately reflect the true anomalies, improving the accuracy of outlier detection.

[0064] Step S103: Determine the weighted density anomaly degree of each suspected outlier in the filtered suspected outlier set, and determine the outliers in the dataset to be detected based on the weighted density anomaly degree of each suspected outlier.

[0065] Weighted density anomaly is an indicator used to measure whether a sample point is an outlier. It combines the local density information of the sample point in the dataset with certain weighting factors. This metric not only considers the local density of the sample point but also introduces additional weighting factors to more accurately reflect the degree of anomaly. When determining the weighted density anomaly of each suspected outlier in the filtered set of suspected outliers, the local density of each suspected outlier is first calculated. Local density can be calculated by considering the neighborhood of the suspected outlier, for example, by calculating the number of points or the distance between the suspected outlier and its neighbors. To more accurately assess the degree of anomaly of suspected outliers, weighting factors can be introduced. These factors can be based on the point's splitting round, the distance from the point to the cluster center, or other relevant attributes. These weighting factors can be used to adjust the calculation results of the local density to better suit specific analytical needs.

[0066] Specifically, the local density of each suspected outlier is calculated by analyzing the density of its neighboring points. Predefined weighting factors, such as the number of splits or other influencing factors, are applied to adjust the local density value. The weighted local density value represents the weighted density anomaly level of the suspected outlier, indicating its degree of anomalousness within its data space, combining local density and weighting information. Based on the weighted density anomaly level of each suspected outlier, it can be further determined whether these points are indeed outliers. An outlier factor threshold can be set, marking suspected outliers with weighted density anomalies exceeding this threshold as outliers. If the weighted density anomaly level of a suspected outlier exceeds the preset threshold, the point is considered anomalous in the dataset, significantly different from most sample points, and therefore identified as an outlier. In this way, through the calculation of weighted density anomalies and threshold determination, true outliers can be accurately identified from the dataset to be detected.

[0067] This invention divides the dataset to be detected into a set of clusters and a set of suspected outliers; filters the suspected outliers in the set of suspected outliers based on the set of clusters; determines the degree of weighted density anomaly of each suspected outlier in the filtered set of suspected outliers, and determines the outliers in the dataset to be detected based on the degree of weighted density anomaly of each suspected outlier; thereby enabling outlier identification during the clustering process, reducing the time complexity of outlier detection, improving detection efficiency, and increasing detection accuracy.

[0068] It should be noted that this embodiment is applicable to various data types, such as image data, text data, time series data, and multidimensional numerical data. That is, the applicability of the detection method is not limited to a specific type of data, but can be flexibly adjusted and optimized according to the characteristics of the data. For example, in image data processing, clustering and outlier detection can be performed on the vectorized representation of image features for the screening and processing of abnormal images; in text data processing, text can be represented as embedded vectors, and outlier detection methods can be applied based on these vectors to identify abnormal semantic content; in time series data, abnormal time periods can be identified by segmenting and clustering sequence segments, and so on.

[0069] Furthermore, this embodiment can optimize the hardware computing efficiency of the computer system and improve overall processing performance. Specifically, splitting and clustering operations can be completed efficiently through parallel computing, especially in multi-core processors or distributed computing environments, which can significantly reduce computing time and improve hardware utilization. At the same time, by calculating the weighted density anomaly degree, it is possible to avoid performing complex global analysis on each suspected outlier, thereby reducing the consumption of computing resources and further improving hardware processing speed.

[0070] In addition to its application in road recognition scenarios, this embodiment can also be applied to many other fields. For example, in the field of financial risk control, it can be used to detect abnormal transactions or potential fraudulent activities; in industrial monitoring, it can be used to detect abnormal operating conditions or malfunctions of equipment; or in recommendation systems, it can be used to filter abnormal user behavior or abnormal item features, etc.

[0071] Optionally, the dataset to be detected is divided into a cluster set and a suspected outlier set, including: splitting the dataset to be detected to obtain multiple clusters; for any cluster: in response to the difference between the number of sample points of any cluster and the preset cluster capacity not exceeding a preset threshold, adding any cluster to the cluster set; in response to the difference between the number of sample points of any cluster and the preset cluster capacity exceeding a preset threshold, adding any cluster to the suspected outlier set.

[0072] A dataset can be divided into several clusters using hierarchical clustering algorithms or splitting strategies. Each cluster represents a subset of data points with similar characteristics, meaning sample points are grouped based on their similarity or features. These clusters are then processed, and their affiliation is determined by the difference between the number of sample points in each cluster and a preset cluster size. The preset cluster size is a standard value used to evaluate cluster size and can be set based on the overall size and feature distribution of the dataset.

[0073] For each cluster, the difference between its sample count and the preset cluster capacity is first calculated. If the difference does not exceed a preset threshold, the cluster is considered normal, its size is within the expected range, and it is added to the cluster set as a normal, expected cluster in the dataset. If the difference exceeds the preset threshold, it indicates a significant difference between the cluster size and the expected capacity. This difference may indicate too many or too few sample counts, or reflect anomalies in the data distribution. The cluster is then added to the suspected outlier set for further analysis as a potential anomalous region.

[0074] Optionally, after dividing the dataset to be detected into a set of clusters and a set of suspected outliers, the method further includes: splitting the clusters in the set of suspected outliers where the sample points are unevenly distributed to obtain two new clusters; comparing the number of sample points in the two new clusters, and removing the sample points in the new cluster with more sample points from the set of suspected outliers.

[0075] Clusters with unevenly distributed sample points within the suspected outlier set are identified, where the number of sample points may vary significantly. These unevenly distributed clusters are then further split, for example, using the FUSHC algorithm for re-clustering or applying data segmentation methods to divide one cluster into two new clusters. After generating these two new clusters, their sample point counts are compared. If one new cluster has significantly more sample points than the other, the data in the larger cluster is considered no longer characteristic of an outlier. Such a larger cluster indicates a more even distribution of sample points and may be part of the normal data. Therefore, the sample points in the new cluster with the larger number of sample points can be removed from the suspected outlier set. This embodiment further cleans up the suspected outlier set through secondary splitting, which can improve the accuracy of anomaly detection and optimize the data classification and processing results.

[0076] Optionally, filtering suspected outliers in the suspected outlier set based on the cluster set includes: for any suspected outlier in the suspected outlier set, determining the cluster in the cluster set that is closest or second closest to any suspected outlier; determining the information entropy difference before and after adding any suspected outlier to the closest or second closest cluster; and removing the suspected outlier from the suspected outlier set in response to the information entropy difference not exceeding a preset change threshold.

[0077] For each suspected outlier in the suspected outlier set, determine the nearest or second-nearest cluster in the cluster set. Specifically, this can be achieved by calculating the distance from the suspected outlier to the center of each cluster, selecting the cluster with the smallest distance as the nearest cluster, and the cluster with the second smallest distance as the second-nearest cluster. Calculate the change in information entropy after adding the suspected outlier to the nearest and second-nearest clusters, respectively. Information entropy measures the degree of uncertainty or disorder in a dataset. Adding a suspected outlier to a cluster may affect the density distribution of the clusters, thus changing their information entropy. By comparing the difference in information entropy before and after adding the suspected outlier, the impact of the suspected outlier on the cluster's information entropy can be evaluated. If the difference in information entropy does not exceed a preset change threshold, it indicates that adding the suspected outlier to the cluster has a small impact on the cluster's information entropy and does not significantly change the overall structure of the cluster. In this case, the suspected outlier can be removed from the suspected outlier set and is no longer considered an abnormal data point.

[0078] Optionally, determining the weighted density anomaly degree of each suspected outlier in the filtered set of suspected outliers includes: determining the splitting round of each suspected outlier in the filtered set of suspected outliers; weighting the original outlier factor of the suspected outlier according to the splitting round to obtain the splitting round weighting factor of the suspected outlier; determining the local density of the suspected outlier based on the splitting round weighting factor; and calculating the weighted density anomaly degree of the suspected outlier based on the splitting round weighting factor and the local density.

[0079] When processing a set of suspected outliers, the accuracy of anomaly detection can be further optimized by analyzing the number of splitting rounds for each suspected outlier. First, the number of splitting rounds for each suspected outlier in the filtered set is determined. The number of splitting rounds refers to the number of times an outlier is re-divided during the splitting process. By recording the number of splitting rounds for each suspected outlier, its stability and changes during the splitting process can be understood. These splitting rounds are then used to weight the original outlier factor for each suspected outlier. The outlier factor is an indicator used to measure the difference between a sample point and its surrounding sample points. Combining the splitting rounds with the original outlier factor and performing a weighted process yields the splitting round weighted factor for the suspected outlier. This weighted factor indicates the degree of influence of the outlier during the splitting process and its sensitivity to the anomalous behavior of the dataset. Based on the split round weighting factor, the local density of each suspected outlier can be further determined. Local density refers to the density of data points within the neighborhood of a suspected outlier, which can be achieved by calculating the number of data points within a certain range around the suspected outlier. Higher local density indicates that the suspected outlier is closer to the normal data distribution within its neighborhood; conversely, lower density suggests that the suspected outlier is more likely to be anomalous. The weighted density anomaly level of each suspected outlier is calculated by combining the split round weighting factor and the local density. This calculation can be performed using a weighted average or other comprehensive methods. The weighted density anomaly level provides the level of anomalousness of a suspected outlier within its neighborhood and considers its influence during the splitting process, thus more accurately assessing its anomalousness. This allows for more effective identification and handling of outliers in the dataset, improving the accuracy and reliability of data analysis.

[0080] Optionally, the original outlier factor of a suspected outlier is obtained through the following steps: in response to the maximum distance between the suspected outlier and the sample points in the nearest cluster being less than the average distance between the suspected outlier and the sample points in the second nearest cluster, the number of neighborhoods of the suspected outlier is determined based on the nearest cluster; in response to the maximum distance between the suspected outlier and the sample points in the nearest cluster being greater than or equal to the average distance between the suspected outlier and the sample points in the second nearest cluster, the number of neighborhoods of the suspected outlier is determined based on the nearest cluster and the second nearest cluster; the original outlier factor is determined based on the number of neighborhoods of the suspected outlier.

[0081] When the maximum distance between a suspected outlier and the sample points in its nearest cluster is less than the average distance between that point and the sample points in its second nearest cluster, the suspected outlier can be considered primarily influenced by its nearest cluster. Therefore, the number of neighborhoods for this suspected outlier is determined based on the number of sample points in the nearest cluster and related information. Conversely, when the maximum distance between a suspected outlier and the sample points in its nearest cluster is greater than or equal to the average distance between that point and the sample points in its second nearest cluster, it indicates that the second nearest cluster also has some influence on the distribution of the suspected outlier. In this case, the characteristics of both the nearest and second nearest clusters need to be considered to determine the number of neighborhoods. Based on the final determined number of neighborhoods, combined with the local density differences between the suspected outlier and its neighbors, the original outlier factor of the suspected outlier can be calculated, thus providing a basis for subsequent anomaly analysis.

[0082] Optionally, determining outliers in the dataset to be detected based on the weighted density anomaly degree of each suspected outlier includes: in response to the weighted density anomaly degree of a suspected outlier exceeding a preset outlier threshold, identifying the suspected outlier as an outlier in the dataset to be detected.

[0083] When processing suspected outliers, their weighted density anomaly level is compared with a preset outlier threshold to determine whether they should be identified as outliers in the dataset to be detected. Specifically, the obtained weighted density anomaly level is compared with a preset outlier threshold, which is set according to the characteristics of the dataset and the analysis objectives, and is used to distinguish between normal data points and abnormal data points. If the weighted density anomaly level of a suspected outlier exceeds this outlier threshold, it indicates that the suspected outlier exhibits significant anomalous characteristics in the dataset and may have a high risk of being an anomaly, and can be identified as an outlier in the dataset to be detected. This indicates that the point no longer conforms to the characteristics of normal data distribution, possibly due to system failure, data errors, or other abnormal factors. Marking these outliers as anomalous data is beneficial for further analysis and processing, such as conducting more in-depth anomaly detection, correcting data quality problems, or taking other corresponding measures. The outlier identification in this embodiment of the invention is based not only on its own characteristics but also on its performance during the data splitting process, thereby improving the accuracy and reliability of anomaly detection.

[0084] Figure 2 This is a schematic diagram of the main flow of an outlier detection method according to a preferred embodiment of the present invention. Figure 2 As shown, the process begins by reading the dataset and using the FUSHC algorithm (Fast and uniform segmentation hierarchical clustering) to perform fast and uniform segmentation. After multiple splits, the size of all clusters is reduced to less than a threshold. Where n is the number of sample points in the dataset. After each round of splitting, it is determined whether the size of the cluster is much smaller than... If not, add the cluster to the cluster set; if yes, add all points in the cluster to the suspected outlier set, save the current rounds (iteration rounds) as their value, update the rounds value, and proceed to the next split until the size of all clusters is less than [the value of the cluster]. until.

[0085] The FUSHC algorithm is used to perform a second split on the density-uniform clusters in the suspected outlier set. For each new cluster formed by the split, all sample points in the new cluster with a significantly smaller sample size than the other cluster are retained in the suspected outlier set, and the current rounds are stored as their value rows. The global suspected outliers quickly selected by the FUSHC algorithm after the two splits are shown below. Figure 3 As shown, the clusters in the cluster set are adaptively merged to form the final cluster set of the dataset. The merging logic is based on the change in the standard deviation of the edge set of the minimum spanning tree of the merged clusters. If the change in the standard deviation of the edge set of the currently stable clusters after merging is greater than a threshold, the two clusters are considered not to belong to the same cluster; otherwise, they are merged. Suspected outliers are taken from the suspected outlier set and sequentially added to their two nearest clusters (the nearest and second nearest clusters). It is then determined whether the cluster information entropy changes significantly after addition, i.e., whether the information entropy change approaches 0. If the cluster information entropy changes significantly, it means that the suspected outlier does not belong to any cluster, and is therefore a suspected outlier in the dataset, and is retained in the suspected outlier set. Conversely, if the information entropy of a certain cluster in cluster A or B does not change significantly after adding a suspected outlier, then the suspected outlier is considered to belong to cluster A or cluster B. Therefore, the suspected outlier is added to the cluster whose information entropy has not changed significantly, and the suspected outlier is removed from the set of suspected outliers.

[0086] For the filtered set of suspected outliers, each suspected outlier is sequentially extracted, and its SRLOF score (Splitting-Rounds Weighted Local Outlier Factor) is calculated based on the weighted distance calculation method for split rounds and the method for determining the number of neighbors k of each suspected outlier. Finally, points with SRLOF scores greater than the outlier factor threshold ξ are selected as the final outliers, and the algorithm terminates.

[0087] Figure 4 This is a schematic diagram of a suspected outlier point between sparse and dense clusters according to a preferred embodiment of the present invention. Figure 5This is a schematic diagram of density-uniform clusters containing suspected outliers according to a preferred embodiment of the present invention. Sample points between sparse and dense clusters are detected using cluster information entropy, assuming the current sparse and dense clusters are as follows: Figure 4 As shown, during the splitting process of the FUSHC algorithm, there exists a special type of data sample point: a sample point that lies between sparse and dense clusters. When the algorithm splits the dataset, it first selects the two points p1 and p2 that are farthest apart in the current cluster, and then allocates other sample points in the cluster according to the nearest neighbor principle, randomly forming clusters A and B. Since the distance from sample point p3 to p1 is less than its distance to p2, p3 is assigned to dense cluster A, which obviously causes p3 to be misclassified. Next, the FUSHC algorithm is used to find the minimum spanning tree of cluster A, as follows... Figure 5 As shown, cluster A clearly belongs to the first type of density-uniform cluster, i.e., a density-uniform cluster connected to the outlier and the dense cluster by the long edge c. The FUSHC algorithm performs a secondary split on this cluster according to the corresponding splitting measures, i.e., disconnecting the long edge c of the partition minimum spanning tree, thereby separating the suspected outlier p3 from the dense cluster A. At this time, the suspected outlier p3 forms a separate cluster. Since the number of sample points in its cluster is much smaller than the other cluster formed after the secondary split of A, p3 is regarded as a suspected outlier. However, through observation, it can be seen that p3 should actually belong to the sparse region B, that is, p3 is a normal point in a sparse cluster that has been misjudged as a suspected outlier. The first stage of outlier detection utilizes cluster information entropy. A suspected outlier, p, is connected to the nearest point in the cluster, essentially adding it as a leaf node to the cluster's minimum spanning tree. The change in the standard deviation of the cluster's minimum spanning tree edge set after this addition is then assessed. If the standard deviation changes significantly, the point does not belong to the cluster and remains in the suspected outlier set. If the standard deviation does not change significantly, the point belongs to the cluster, is added to the cluster, and the suspected outlier is removed from the suspected outlier set. Using changes in cluster information entropy to identify suspected outliers effectively addresses the misclassification of normal data points that fall between dense and sparse clusters.

[0088] When performing a secondary split on clusters with uneven density, the suspected outliers may be local outliers in the dataset, or normal points between sparse and dense clusters that have been mistakenly identified as suspected outliers. Therefore, a smaller weight can be assigned to these two types of suspected outliers when calculating the distance. This embodiment also proposes an adaptive method for determining k, solving the problem of choosing the number of neighborhoods k in the parameters of the LOF algorithm. The definitions and concepts involved in solving the local anomaly factor for the split round weighting of suspected outliers are given below.

[0089] Definition 1: The k-th distance (k-distance(p)) of a suspected outlier p. Let d(p, o) represent the distance from data point o to the suspected outlier p. Then, k-distance(p) is the k-th distance between p and all sample points, sorted from smallest to largest. In this algorithm, k is adaptively determined based on the size of the nearest cluster C1 and the second nearest cluster C2 of the suspected outlier, and the dimensions of the dataset. It does not need to be given in advance as an input parameter of the algorithm. The k-nearest neighbor count of a suspected outlier is calculated as follows:

[0090]

[0091] In the above formula: N C1 N is the number of sample points in cluster C1 that is closest to the suspected outlier p; C2 It is the number of sample points in cluster C2 that is p closest to the suspected outlier; d is the data dimension; d max (C1, p) is the maximum distance from the suspected outlier point to cluster C1; d avg (C2, p) is the average distance of the suspected outlier from cluster C2.

[0092] From the formula for calculating k, we know that if the maximum distance between a suspected outlier and the midpoint of its nearest cluster is less than the average distance to its second nearest cluster, then the number of its k-nearest neighbors is determined solely by the size of the nearest cluster C1. Conversely, if the maximum distance between a suspected outlier and the midpoint of its nearest cluster is greater than or equal to the average distance to its second nearest cluster, then the value of k is determined by the sizes of its nearest and second nearest clusters. The more data points in a cluster, the larger the value of k; conversely, the smaller the number of data points, the smaller the value of k. Similarly, the value of k is directly proportional to the dimension.

[0093] Definition 2: Split-round weighted distance. Assume p, o ∈ D, and p is a suspected outlier, o is a data point in the cluster. Then the split-round weighted distance between objects p and o is expressed as:

[0094]

[0095] In the above formula: x di (p) and x di (q) is the value of the i-th (i = 1, 2, ..., d)-th attribute; ω round It is the weight of the suspected outlier points split out in the i-th round of the algorithm.

[0096] ω round The solution method is as follows:

[0097]

[0098] In the formula: round is the number of separation processes in which the current suspected outlier is obtained; λ is the weighting coefficient, which is used to weigh the weight added by the algorithm in the weighted distance calculation of the split round.

[0099] The method for determining the number of k nearest neighbors of a suspected outlier point proposed in Definition 1 and the distance calculation method based on split round weighting proposed in Definition 2 are applied to the following formula. The k-th reachable distance from point o to the suspected outlier point p is defined as:

[0100] reach-dist k (p,o,ω)=max{k-distance(o),d(p,o,ω)}

[0101] In the above formula: k-distance(o) is the k-th distance of sample point o; max is the maximum value function; reach-distk(p, o, ω) is the k-th reachable distance from sample point o to sample point p.

[0102] The local reachability density calculated using the split-cycle weighted distance of the suspected outlier point p is as follows:

[0103]

[0104] In the above formula: |Nk(p)| is the number of k-nearest neighbors of sample point p; SRlrd k (p) is the local reachability density calculated by the split-cycle weighted distance of the suspected outlier point p.

[0105] Therefore, the LOF calculation formula for suspected outliers based on split-round weighted distance is as follows:

[0106]

[0107] In the above formula: |Nk(p)| is the number of k-nearest neighbors of sample point p; SRlrd k (p) is the local reachability density calculated by the split-cycle weighted distance of the suspected outlier point p.

[0108] By using a distance calculation method that weights the splitting rounds, the distance between local outliers and normal data points in the cluster can be increased. This is because sample points split earlier in the FUSHC algorithm's splitting process are more likely to be true outliers in the dataset. Weighting these points according to their splitting order makes the calculation of their local outlier factors more accurate.

[0109] Figure 6 This is the original image obtained by applying outlier detection according to a preferred embodiment of the present invention. Figure 7 This is a schematic diagram illustrating the detection effect of outlier detection according to a preferred embodiment of the present invention. Figure 6 As shown and Figure 7 As shown, thumbnails representing the dataset are obtained, and feature vectors of the images are extracted using a color histogram-based feature extraction method. For two 720×1280 images, outlier detection algorithms CBLOF (Clustering-Based Local Outlier Factor), KMOR (K-Medoids Outlier Ranking), and ODC (Outlier Detection with Clustering) are used to identify outliers in the images. Then, the SRLOF algorithm proposed in this embodiment is used to detect outliers on the same image, and the detection effects of different algorithms on the same image are observed.

[0110] like Figure 6 As shown, using images of the library's roads as the dataset, this study detects outliers other than the roads themselves. Photos of the road behind the library were taken, and the images consist of roads, grass and bushes (normal points in the image), with a small number of walls and trees (outliers). The images mainly contain seven categories: roads, grass, bushes, trees, walls, buildings, and other miscellaneous objects. The outlier detection results are used to judge the effectiveness of the detection by distinguishing between a large number of primary elements and a small number of secondary elements in the original image. Roads and bushes are considered primary elements, while trees, walls, and distant buildings are considered secondary elements.

[0111] like Figure 7 As shown in the image, the detection results of the four outlier detection algorithms are analyzed. The SRLOF algorithm can accurately distinguish roads, lawns, trees, and walls, and can also identify soil in lawns and shadows on roads as local outliers. While the CBLOF algorithm can effectively identify roads, grass, and shadows on roads, it performs poorly in identifying trees and walls within grass. The ODC algorithm can identify roads and grass, but its performance is poor in identifying walls between grass. The KMOR algorithm only identifies roads and not grass, making it the worst at identifying outliers in the image.

[0112] According to a second aspect of the present invention, an outlier detection device is provided.

[0113] Figure 8 This is a schematic diagram of the main modules of an outlier detection device according to an embodiment of the present invention. Figure 8 As shown, the outlier detection device 800 includes:

[0114] The partitioning module 801 is used to divide the dataset to be detected into a set of clusters and a set of suspected outliers.

[0115] Filtering module 802 is used to filter suspected outliers in the suspected outlier set based on the cluster set;

[0116] The determination module 803 is used to determine the weighted density anomaly degree of each suspected outlier in the filtered suspected outlier set, and to determine the outliers in the dataset to be detected based on the weighted density anomaly degree of each suspected outlier.

[0117] Optionally, the partitioning module 801 is also used for:

[0118] The dataset to be detected is split to obtain multiple clusters;

[0119] For any cluster among multiple clusters: if the difference between the number of sample points of any cluster and the preset cluster capacity does not exceed a preset threshold, add any cluster to the cluster set; if the difference between the number of sample points of any cluster and the preset cluster capacity exceeds a preset threshold, add any cluster to the suspected outlier set.

[0120] Optionally, the detection device 800 further includes a re-division module, which is used for:

[0121] The clusters with unevenly distributed sample points in the suspected outlier set are split again to obtain two new clusters;

[0122] Compare the number of sample points in the two new clusters, and remove the sample points in the new cluster with more sample points from the suspected outlier set.

[0123] Optionally, the filter module 802 is also used for:

[0124] For any suspected outlier in the set of suspected outliers, determine the cluster in the cluster set that is closest to or second closest to any suspected outlier;

[0125] Determine the difference in information entropy before and after adding any suspected outlier to the nearest or second nearest cluster;

[0126] If the information entropy difference does not exceed the preset change threshold, suspected outliers are removed from the suspected outlier set.

[0127] Optionally, the determining module 803 is also used for:

[0128] Determine the splitting round of each suspected outlier in the filtered set of suspected outliers, and weight the original outlier factor of the suspected outlier according to the splitting round to obtain the splitting round weighting factor of the suspected outlier.

[0129] The local density of suspected outliers is determined based on the splitting round weighting factor of suspected outliers;

[0130] The weighted density anomaly degree of suspected outliers is calculated based on the splitting round weighting factor and local density.

[0131] Optionally, the detection device 800 further includes an adaptation module, which is used for:

[0132] If the maximum distance between a suspected outlier and a sample point in the nearest cluster is less than the average distance between a suspected outlier and a sample point in the second nearest cluster, then the number of neighbors of the suspected outlier is determined based on the nearest cluster.

[0133] If the maximum distance between the suspected outlier and the sample point in the nearest cluster is greater than or equal to the average distance between the suspected outlier and the sample point in the second nearest cluster, then the number of neighborhoods of the suspected outlier is determined based on the nearest cluster and the second nearest cluster.

[0134] The original outlier factor is determined based on the number of neighborhoods of suspected outliers.

[0135] Optionally, the determining module 803 is also used for:

[0136] If the weighted density anomaly of a suspected outlier exceeds a preset outlier threshold, the suspected outlier is identified as an outlier in the dataset to be detected.

[0137] It should be noted that the specific implementation details of the outlier detection device of the present invention have been described in detail in the outlier detection method above, so the details will not be repeated here.

[0138] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the present invention.

[0139] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0140] According to a fifth aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0141] Figure 9 An exemplary system architecture 900 is shown that can be applied to the outlier detection method or outlier detection apparatus of the present invention.

[0142] like Figure 9As shown, system architecture 900 may include terminal devices 901, 902, and 903, network 904, and server 905. Network 904 is used as a medium to provide a communication link between terminal devices 901, 902, and 903 and server 905. Network 904 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0143] Users can use terminal devices 901, 902, and 903 to interact with server 905 via network 904 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 901, 902, and 903, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0144] Terminal devices 901, 902, and 903 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0145] Server 905 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 901, 902, and 903 (for example only). The backend management server can analyze and process data such as received outlier detection requests, and feed back the processing results (such as outlier detection results - for example only) to the terminal devices.

[0146] It should be noted that the outlier detection method provided in this embodiment of the invention is generally run by server 905, and correspondingly, the outlier detection device is generally set in server 905.

[0147] It should be understood that Figure 9 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0148] The following is for reference. Figure 10 It shows a schematic diagram of the structure of a computer system 1000 suitable for implementing a terminal device of the present invention. Figure 10 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0149] like Figure 10As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1002 or programs loaded from storage section 1008 into random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the system 1000. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0150] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed.

[0151] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is run by central processing unit (CPU) 1001, it performs the functions defined above in the system of this invention.

[0152] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more operable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually operate substantially in parallel, and they may sometimes operate in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0154] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a determination module, a filtering module, and a partitioning module. The names of these modules do not necessarily limit the module itself; for example, the partitioning module can also be described as "a module for dividing the dataset to be detected into a set of clusters and a set of suspected outliers."

[0155] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to include: dividing a dataset to be detected into a set of clusters and a set of suspected outliers; filtering suspected outliers in the set of suspected outliers based on the set of clusters; determining the weighted density anomaly degree of each suspected outlier in the filtered set of suspected outliers; and determining outliers in the dataset to be detected based on the weighted density anomaly degree of each suspected outlier.

[0156] The computer program product provided in this embodiment of the invention includes a computer program that, when executed by a processor, implements the outlier detection method in this embodiment of the invention.

[0157] According to the technical solution of the present invention, the following advantages or beneficial effects are achieved: by dividing the dataset to be detected into a set of clusters and a set of suspected outliers; filtering the suspected outliers in the set of suspected outliers based on the set of clusters; determining the weighted density anomaly degree of each suspected outlier in the filtered set of suspected outliers; and determining the outliers in the dataset to be detected based on the weighted density anomaly degree of each suspected outlier; thereby reducing the time complexity of outlier detection, improving detection efficiency, and increasing detection accuracy.

[0158] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0159] It should be noted that the acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

Claims

1. A method for outlier detection, characterized in that, include: The dataset to be detected is divided into a set of clusters and a set of suspected outliers; Based on the cluster set, filter the suspected outliers in the suspected outlier set; Determine the degree of weighted density anomaly of each suspected outlier in the filtered set of suspected outliers, and determine the outliers in the dataset to be detected based on the degree of weighted density anomaly of each suspected outlier.

2. The method according to claim 1, characterized in that, The dataset to be detected is divided into a set of clusters and a set of suspected outliers, including: The dataset to be detected is split to obtain multiple clusters; For any one of the multiple clusters: if the difference between the number of sample points of any one cluster and the preset cluster capacity does not exceed a preset threshold, add any one cluster to the cluster set; if the difference between the number of sample points of any one cluster and the preset cluster capacity exceeds a preset threshold, add any one cluster to the suspected outlier set.

3. The method according to claim 1, characterized in that, After dividing the dataset to be detected into a set of clusters and a set of suspected outliers, it also includes: The clusters with unevenly distributed sample points in the suspected outlier set are split again to obtain two new clusters; By comparing the number of sample points in the two new clusters, the sample points in the new cluster with the larger number of sample points are removed from the set of suspected outliers.

4. The method according to claim 1, characterized in that, Filtering suspected outliers in the suspected outlier set based on the cluster set includes: For any suspected outlier in the set of suspected outliers, determine the cluster in the set of clusters that is closest to or second closest to any suspected outlier; Determine the difference in information entropy before and after adding any suspected outlier to the nearest or second nearest cluster; In response to the information entropy difference not exceeding a preset change threshold, the suspected outlier is removed from the suspected outlier set.

5. The method according to claim 1, characterized in that, Determine the weighted density anomaly degree of each suspected outlier in the filtered set of suspected outliers, including: Determine the splitting round of each suspected outlier in the filtered set of suspected outliers, and weight the original outlier factor of the suspected outlier according to the splitting round to obtain the splitting round weighting factor of the suspected outlier. The local density of the suspected outliers is determined based on the splitting round weighting factor of the suspected outliers; The weighted density anomaly degree of the suspected outlier is calculated based on the weighting factor of the splitting round and the local density.

6. The method according to claim 5, characterized in that, The original outlier factors of the suspected outliers are obtained through the following steps: If the maximum distance between the suspected outlier and the sample point in the nearest cluster is less than the average distance between the suspected outlier and the sample point in the second nearest cluster, then the number of neighbors of the suspected outlier is determined based on the nearest cluster. In response to the maximum distance between the suspected outlier and the sample point in the nearest cluster being greater than or equal to the average distance between the suspected outlier and the sample point in the second nearest cluster, the number of neighborhoods of the suspected outlier is determined based on the nearest cluster and the second nearest cluster. The original outlier factor is determined based on the number of neighborhoods of the suspected outlier.

7. The method according to claim 1, characterized in that, Determining outliers in the dataset to be detected based on the weighted density anomaly degree of each suspected outlier includes: In response to the weighted density anomaly of the suspected outlier exceeding a preset outlier threshold, the suspected outlier is identified as an outlier in the dataset to be detected.

8. An outlier detection device, characterized in that, include: The partitioning module is used to divide the dataset to be detected into a set of clusters and a set of suspected outliers; The filtering module is used to filter suspected outliers in the suspected outlier set based on the cluster set; The determination module is used to determine the weighted density anomaly degree of each suspected outlier in the filtered set of suspected outliers, and to determine the outliers in the dataset to be detected based on the weighted density anomaly degree of each suspected outlier.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images