Abnormal data detection method and apparatus, and storage medium

By dividing the telecommunications flow data into multiple data sets for clustering, and clustering centers, eventually selected abnormal data in each cluster, solving the high complexity in the existing technology The problem has achieved efficient abnormal data detection.

WO2025016349A1PCT designated stage expired Publication Date: 2025-01-23CHINA TELECOM CORP LTD TECHNOLOGY INNOVATION CENTER +1

Patent Information

Application Number
PCT/CN2024/105401
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-07-14
Filing Date
2024-07-15
Publication Date
2025-01-23

AI Technical Summary

Technical Problem

The existing dynamic clustering algorithm has high computational complexity in telecommunications traffic data, slow network convergence speed, and clustering results are greatly affected by threshold values, making it difficult to accurately detect abnormal data.

Method used

By dividing the preset data set into multiple data sets, each data set is used for clustering, multiple clustering results are obtained, and the cluster center is used for clustering. Finally, according to the data and cluster center in each cluster Select abnormal data.

Benefits of technology

It effectively reduces the computational complexity, improves the accuracy and efficiency of abnormal data detection, and does not require manual setting of the number of clusters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024105401_23012025_PF_FP_ABST
    Figure CN2024105401_23012025_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure provides an abnormal data detection method and apparatus, and a storage medium. The abnormal data detection method comprises: segmenting a preset data set into multiple data subsets, the data volume of each data subset being less than a preset data volume threshold; clustering each data subset, to obtain multiple clustering results; performing clustering of cluster centers in the multiple clustering results, so as to divide multiple pieces of data in the preset data set into multiple clusters; and in each cluster among the multiple clusters, selecting abnormal data on the basis of the distance between each piece of data and the cluster center.
Need to check novelty before this filing date? Find Prior Art

Description

Abnormal data detection method and device, and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure is based on and claims priority to an application with CN application number 202310869053.6 and filing date July 14, 2023. The disclosure content of this CN application is hereby incorporated into this disclosure as a whole. Technical Field

[0003] The present disclosure relates to the field of data mining, and in particular to a method and device for detecting abnormal data, and a storage medium. Background Art

[0004] As telecom operators' businesses continue to expand, the volume of traffic on their servers is growing exponentially. Selecting non-anomalous data from this vast and complex data set for analysis is a challenging task. Therefore, it is essential to extract features from telecom traffic data, minimize data dimensions, ensure accurate classification, and perform outlier detection.

[0005] The existing dynamic clustering algorithm does not need to set the number of categories in advance. It is a clustering algorithm that can automatically find the number of categories. It is described as follows:

[0006] Step 1: Build an undirected graph. For each node x i , assign the initial category class(x i )=i;Calculate the similarity between different nodes. If the similarity is greater than the preset threshold, build a related edge and set the weight value to the similarity value.

[0007] Step 2: Traverse the process. Randomly select x i If the nodes in the neighborhood belong to the same category, add up the weights of these nodes in sequence; select the largest category m as the current number of categories.

[0008] Step 3: Repeat step 2 until the traversal is completed and the number of classification clusters k is obtained.

[0009] Summary of the Invention

[0010] In a first aspect of the present disclosure, a method for detecting abnormal data is provided, comprising: dividing a preset data set into multiple data subsets, wherein the data amount of each data subset is less than a preset data amount threshold; clustering each data subset to obtain multiple clustering results; clustering the cluster centers in the multiple clustering results so as to divide the multiple data in the preset data set into multiple clusters; and selecting abnormal data in each cluster of the multiple clusters based on the distance between each data and the cluster center.

[0011] In some embodiments, in each of the plurality of clusters, selecting abnormal data according to the distance between each data and the cluster center includes: calculating the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster. i , where 1≤m≤M, M is the total number of clusters, 1≤i≤N, N is the total number of data in the m-th cluster; according to the distance D i Determine the outlier value of the i-th data; if the outlier value of the i-th data is greater than a preset outlier threshold, determine that the i-th data in the m-th cluster is outlier data.

[0012] In some embodiments, the outlier value of the i-th data is the distance D i .

[0013] In some embodiments, according to the distance D i Determining the abnormal value of the i-th data includes: according to the distance D i The outlier value of the i-th data is determined by summing the local density of the i-th data.

[0014] In some embodiments, the outlier value of the i-th data is the distance D i and the ratio of the local density of the i-th data.

[0015] In some embodiments, the local density of the i-th data is determined by the distance between the i-th data and the j-th data in the m-th cluster, where 1≤j≤N and j≠i.

[0016] In some embodiments, the local density ρ of the i-th data i for

[0017] where dist(x i , x j ) is the i-th data x i The jth data x j The distance between c is the preset cutoff distance.

[0018] In a second aspect of the present disclosure, an abnormal data detection device is provided, comprising: a first processing module, configured to divide a preset data set into multiple data subsets, wherein the data amount of each data subset is less than a preset data amount threshold; a second processing module, configured to cluster each of the data subsets to obtain multiple clustering results; a third processing module, configured to cluster the cluster centers in the multiple clustering results so as to divide the multiple data in the preset data set into multiple clusters; and a fourth processing module, configured to select abnormal data in each cluster of the multiple clusters based on the distance between each data and the cluster center.

[0019] In some embodiments, the fourth processing module is configured to calculate the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster. i , where 1≤m≤M, M is the total number of clusters, 1≤i≤N, N is the total number of data in the mth cluster, according to the distance D i An outlier value of the i-th data is determined, and if the outlier value of the i-th data is greater than a preset outlier threshold, the i-th data in the m-th cluster is determined to be outlier data.

[0020] In some embodiments, the outlier value of the i-th data is the distance D i .

[0021] In some embodiments, the fourth processing module is configured to process the data according to the distance D i The outlier value of the i-th data is determined by summing the local density of the i-th data.

[0022] In some embodiments, the outlier value of the i-th data is the distance D i and the ratio of the local density of the i-th data.

[0023] In some embodiments, the local density of the i-th data is determined by the distance between the i-th data and the j-th data in the m-th cluster, where 1≤j≤N and j≠i.

[0024] In some embodiments, the local density ρ of the i-th data i for

[0025] where dist(x i , x j ) is the i-th data x i The jth data x j The distance between c is the preset cutoff distance.

[0026] In a third aspect of the present disclosure, an abnormal data detection device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the method as described in any of the above embodiments based on instructions stored in the memory.

[0027] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method described in any of the above embodiments is implemented.

[0028] According to a fifth aspect of an embodiment of the present disclosure, a computer program is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, the method described in any of the above embodiments is implemented.

[0029] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0031] FIG1 is a schematic diagram of a flow chart of an abnormal data detection method according to an embodiment of the present disclosure;

[0032] FIG2 is a schematic structural diagram of an abnormal data detection device according to an embodiment of the present disclosure;

[0033] FIG3 is a schematic structural diagram of an abnormal data detection device according to another embodiment of the present disclosure;

[0034] Figure 4 is a schematic diagram of the time comparison of different clustering methods;

[0035] Figure 5(A)-Figure 5(D) are schematic diagrams of the AUC of different anomaly detection methods on four datasets;

[0036] Figure 6(A)-Figure 6(D) are schematic diagrams of the RMSE of different anomaly detection methods on four datasets. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0038] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0039] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0040] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.

[0041] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0042] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0043] The inventors have noticed that in related technologies, dynamic clustering algorithms have the following deficiencies:

[0044] 1. In telecommunications traffic data, the edge weights in the network are unknown, so prior knowledge is required to obtain the edge weights. Since building a connection matrix for a small dataset requires calculating the similarity between nodes, the time complexity is high and the network convergence speed is slow.

[0045] 2. The clustering results will be affected by the threshold value, and it is crucial to effectively select the appropriate threshold value.

[0046] Accordingly, the present disclosure provides a solution for detecting abnormal data, which can effectively reduce computational complexity and accurately implement abnormal data detection.

[0047] Figure 1 is a flow chart of an abnormal data detection method according to an embodiment of the present disclosure. In some embodiments, the following abnormal data detection method is performed by an abnormal data detection device.

[0048] In step 101 , a preset data set is divided into a plurality of data subsets, wherein the data volume of each data subset is smaller than a preset data volume threshold.

[0049] It should be noted here that by limiting the amount of data in each data subset, the performance of the clustering algorithm can be ensured while ensuring the stability of the clustering results of each data subset.

[0050] In step 102, clustering is performed on each data subset to obtain a plurality of clustering results.

[0051] In step 103, cluster centers in the plurality of clustering results are clustered so as to divide the plurality of data in the preset data set into a plurality of clusters.

[0052] For example, the preset data set is divided into two data subsets. The clustering result of the first data subset has two clusters, whose centers are a1 and b1 respectively. The clustering result of the second data subset has two clusters, whose centers are a2 and c1 respectively. By clustering a1, b1, a2 and c1, three clusters are obtained, of which the first cluster includes a1 and a2, the second cluster includes b1, and the third cluster includes c1. In this case, the data originally included in the cluster with a1 as the center and the data with a2 as the center are marked as the first category, the data originally included in the cluster with b1 as the center is marked as the second category, and the data originally included in the cluster with c1 as the center is marked as the third category.

[0053] In step 104 , in each of the plurality of clusters, abnormal data is selected based on the distance between each data and the cluster center.

[0054] In some embodiments, the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster is calculated. i , where 1≤m≤M, M is the total number of clusters, 1≤i≤N, N is the total number of data in the mth cluster, according to the distance D i Determine the outlier value of the i-th data. If the outlier value of the i-th data is greater than the preset outlier threshold, determine that the i-th data in the m-th cluster is outlier data.

[0055] For example, the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster is i As shown in formula (1). i =dist(x im , C m )=||x im -C m ‖ (1)

[0056] Among them, x im is the i-th data in the m-th cluster, C m is the cluster center of the mth cluster.

[0057] In some embodiments, the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster is directly calculated. i As the outlier of the i-th data. That is, the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster i The larger the value is, the greater the probability that the i-th data is an outlier.

[0058] In some embodiments, considering that there may be densely distributed data at the boundary between clusters, if only the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster is considered, iIn order to avoid misjudging non-abnormal data as abnormal data, according to the distance D i The local density of the ith data determines the outlier of the ith data.

[0059] In some embodiments, the outlier value of the i-th data is the distance D i and the ratio of the local density of the i-th data.

[0060] For example, the outlier score (x i ) as shown in formula (2).

[0061] Among them, ρ i is the local density of the i-th data.

[0062] From formula (2), we can see that the outlier value of the i-th data is related to dist(x im , C m ) is proportional to ρ i In other words, data with low local density will be regarded as abnormal data with a high probability.

[0063] In some embodiments, the i-th data p i The local density of is determined by the distance between the i-th data and the j-th data in the m-th cluster, where 1≤j≤N and j≠i.

[0064] For example, the local density ρ of the i-th data i It is generated using the Gaussian kernel function, as shown in formula (3).

[0065] Among them, dist(x i , x j ) is the i-th data x i The jth data x j The distance between c is the preset cutoff distance.

[0066] For example, the data in each cluster can be sorted according to a predetermined rule, and an appropriate value can be selected as the cutoff distance d according to the distance between adjacent data. c .

[0067] It should be noted here that the existing K-Means and K-Means++ clustering algorithms are random when selecting the initial cluster centers, which leads to unstable clustering results, easy to fall into local optimal solutions, and a large number of iterations. In addition, the number of clusters needs to be manually set. In the abnormal data detection method provided in the above embodiment of the present disclosure, the preset data set is divided into multiple data subsets, each data subset is clustered to obtain multiple clustering results, and the cluster centers in the multiple clustering results are clustered so as to divide the multiple data in the preset data set into multiple clusters. Then, in each cluster, abnormal data is selected based on the distance between each data and the cluster center. In other words, the present disclosure can effectively reduce the computational complexity by clustering each data subset instead of directly clustering the data set. At the same time, the present application does not need to manually set the number of clusters and can accurately realize abnormal data detection.

[0068] FIG2 is a schematic diagram of the structure of an abnormal data detection device according to an embodiment of the present disclosure. As shown in FIG2 , the abnormal data detection device includes a first processing module 21 , a second processing module 22 , a third processing module 23 and a fourth processing module 24 .

[0069] The first processing module 21 is configured to divide a preset data set into a plurality of data subsets, wherein the data volume of each data subset is smaller than a preset data volume threshold.

[0070] It should be noted here that by limiting the amount of data in each data subset, the performance of the clustering algorithm can be ensured while ensuring the stability of the clustering results of each data subset.

[0071] The second processing module 22 is configured to cluster each data subset to obtain a plurality of clustering results.

[0072] The third processing module 23 is configured to cluster the cluster centers in the plurality of clustering results so as to divide the plurality of data in the preset data set into a plurality of clusters.

[0073] The fourth processing module 24 is configured to select abnormal data in each of the plurality of clusters according to the distance between each data and the cluster center.

[0074] In some embodiments, the fourth processing module 24 is configured to calculate the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster. i , where 1≤m≤M, M is the total number of clusters, 1≤i≤N, N is the total number of data in the mth cluster, according to the distance D i Determine the outlier value of the i-th data. If the outlier value of the i-th data is greater than the preset outlier threshold, determine that the i-th data in the m-th cluster is outlier data.

[0075] For example, the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster is i As shown in formula (1).

[0076] In some embodiments, the fourth processing module 24 directly calculates the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster. i As the outlier of the i-th data. That is, the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster i The larger the value is, the greater the probability that the i-th data is an outlier.

[0077] In some embodiments, the fourth processing module 24 is configured to process the i The local density of the ith data determines the outlier of the ith data.

[0078] In some embodiments, the outlier value of the i-th data is the distance D i and the ratio of the local density of the i-th data.

[0079] For example, the outlier score (x i ) as shown in formula (2).

[0080] In some embodiments, the local density of the i-th data is determined by the distance between the i-th data and the j-th data in the m-th cluster, where 1≤j≤N and j≠i.

[0081] For example, the local density ρ of the i-th data i It is generated using the Gaussian kernel function, as shown in formula (3).

[0082] FIG3 is a schematic diagram of the structure of an abnormal data detection device according to another embodiment of the present disclosure. As shown in FIG3 , the abnormal data detection device includes a memory 31 and a processor 32 .

[0083] The memory 31 is used to store instructions. The processor 32 is coupled to the memory 31 . The processor 32 is configured to execute the method involved in any embodiment in FIG. 1 based on the instructions stored in the memory.

[0084] As shown in Figure 3, the abnormal data detection device also includes a communication interface 33 for exchanging information with other devices. At the same time, the abnormal data detection device also includes a bus 34, through which the processor 32, the communication interface 33, and the memory 31 communicate with each other.

[0085] Memory 31 may include high-speed RAM memory or non-volatile memory, such as at least one disk drive. Memory 31 may also be a memory array. Memory 31 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0086] Furthermore, the processor 32 may be a central processing unit (CPU), or may be an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the present disclosure.

[0087] The present disclosure also relates to a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method involved in any embodiment of FIG. 1 is implemented.

[0088] The present disclosure is described below through specific examples.

[0089] Traffic data is a typical example of time series data, providing valuable insights for system performance management, risk prevention, and anomaly detection. The experimental data, collected from a telecommunications branch from September 1 to October 15, 2022, was used for experimental training, testing, and validation. Data was sampled every five minutes, yielding a total of 1,200 data points. First, the data was preprocessed, cleaned, and filtered to identify 880 valid data sets. 600 of these were used for model training, and the remaining 280 were used to test the network model's performance.

[0090] In order to evaluate the effectiveness of the anomaly detection algorithm proposed in this disclosure, three evaluation indicators are used: Average Precision AP (Average Precision), F1max and the closed area AUC (Area Under Curve) obtained by the observer operating characteristic curve. Assume that M h and M q They represent the true data and q candidate anomaly values ​​respectively, and the precision value is expressed as the ratio of the true anomaly data to the candidate anomaly data:

[0091] The average precision AP is expressed as the average of all values ​​of P(q).

[0092] The recall rate is expressed as formula (5):

[0093] P(q) represents the state in which a true outlier exists among the candidate outliers, and R(q) represents the state in which a true outlier is successfully detected. F1(q) represents the harmonic mean of P(q) and R(q), which is defined as follows:

[0094] F1max refers to the maximum value of F1(q) among q outliers.

[0095] The evaluation indicator disclosed herein uses AUC, which only requires calculating the area under the ROC (receiver operating characteristic curve), whose value is between [0, 1]. A larger value represents a better abnormality detection effect.

[0096] In order to verify the time efficiency of the improved dynamic clustering algorithm proposed in this disclosure, the K-Means algorithm and the K-Means++ algorithm are used for clustering respectively. The experimental results are shown in Figure 4. As shown in Figure 3, when the data scale increases, the clustering time of the K-Means algorithm and the K-Means++ algorithm shows a quadratic growth trend, while the clustering time of the present invention shows a linear growth trend. It can be seen that the solution provided by this disclosure has greatly improved the time efficiency, and the time complexity has been reduced from the original O(n 2 ) becomes O(np).

[0097] In order to analyze the efficiency of the anomaly detection algorithm proposed in this disclosure, this disclosure uses the anomaly detection algorithm proposed in this disclosure and other algorithms to perform performance testing and evaluation in four real data sets S1, S2, S3 and S4. These algorithms are: 3sigma, KNN (K Nearest Neighbors), SOS (Stochastic Outlier Selection) and PCA (Principal Component Analysis). The average of the three experimental results of the next 20 minutes, 40 minutes and 60 minutes is taken. The data is detailed in Table 1, and the experimental results are detailed in Figures 5(A) to 5(D).

[0098] Table 1 Comparison of model prediction performance

[0099] As can be seen from Table 1, regardless of the training time of 20 minutes, 40 minutes, or 60 minutes, the AUC of the anomaly detection algorithm proposed in this disclosure is higher than that of other anomaly detection algorithms. Taking the data from the S2 dataset at 60 minutes as an example, the AUC of the proposed algorithm is 26.3%, 22.9%, 35.8%, and 2.2% higher than those of 3sigma, KNN, SOS, and PCA, respectively. This shows that the anomaly detection algorithm proposed in this disclosure has good overall accuracy.

[0100] Figure 6 (A) to Figure 6 (D) show the results of the prediction performance of the model over time. As the prediction time becomes longer, the prediction error of the model will gradually increase, and the prediction effect will inevitably deteriorate. The RMSE (Root mean squared error) values ​​of 3sigma, KNN, SOS and PCA continue to increase. Compared with these four models, the RMSE variation range of the model proposed in the present disclosure is relatively small, indicating that the overall performance is better. In Figure 6 (A) to Figure 6 (D), mark 1 represents the result of the 3sigma algorithm, mark 2 represents the result of the KNN algorithm, mark 3 represents the result of the SOS algorithm, mark 4 represents the result of the PCA algorithm, and mark 5 represents the result of the solution provided in the present disclosure.

[0101] In some embodiments, the functional units described above may be implemented as general-purpose processors, programmable logic controllers (PLC), digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any appropriate combination thereof, for performing the functions described in the present disclosure.

[0102] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0103] The description of the present disclosure is provided for purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the disclosed form. Many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure and to enable those skilled in the art to understand the present disclosure and design various embodiments with various modifications suitable for specific applications.

Claims

1. A method for detecting abnormal data, comprising: Dividing a preset data set into multiple data subsets, wherein the data volume of each data subset is less than a preset data volume threshold; Clustering each of the data subsets to obtain multiple clustering results; Clustering the cluster centers in the plurality of clustering results so as to divide the plurality of data in the preset data set into a plurality of clusters; In each of the plurality of clusters, abnormal data is selected according to the distance between each data and the cluster center.

2. The method according to claim 1, wherein: In each of the plurality of clusters, selecting abnormal data according to the distance between each data and the cluster center includes: Calculate the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster i , where 1≤m≤M, M is the total number of clusters, 1≤i≤N, N is the total number of data in the mth cluster; According to the distance D i Determine the outlier value of the i-th data; If the abnormal value of the i-th data is greater than a preset abnormal value threshold, the i-th data in the m-th cluster is determined to be abnormal data.

3. The method according to claim 2, wherein: The outlier value of the i-th data is the distance D i .

4. The method according to claim 2, wherein: According to the distance D i Determining the outlier value of the i-th data includes: According to the distance D i And the local density of the i-th data determines the outlier of the i-th data.

5. The method according to claim 4, wherein: The outlier value of the i-th data is the distance D i and the ratio of the local density of the i-th data.

6. The method according to claim 4 or 5, wherein: The local density of the i-th data is the sum of the i-th data and the j-th data in the m-th cluster. The distance is determined where 1≤j≤N and j≠i.

7. The method according to claim 6, wherein: The local density ρ of the i-th data i for where dist(x i , x j ) is the i-th data x i The jth data x j The distance between c is the preset cutoff distance.

8. An abnormal data detection device, comprising: A first processing module is configured to divide a preset data set into a plurality of data subsets, wherein the data volume of each data subset is less than a preset data volume threshold; A second processing module is configured to cluster each of the data subsets to obtain a plurality of clustering results; A third processing module is configured to cluster the cluster centers in the plurality of clustering results so as to divide the plurality of data in the preset data set into a plurality of clusters; The fourth processing module is configured to select abnormal data in each of the multiple clusters according to the distance between each data and the cluster center.

9. The device according to claim 8, wherein: The fourth processing module is configured to calculate the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster. i , where 1≤m≤M, M is the total number of clusters, 1≤i≤N, N is the total number of data in the mth cluster, according to the distance D i An outlier value of the i-th data is determined, and if the outlier value of the i-th data is greater than a preset outlier threshold, the i-th data in the m-th cluster is determined to be outlier data.

10. The device according to claim 9, wherein: The outlier value of the i-th data is the distance D i .

11. The device according to claim 9, wherein: The fourth processing module is configured to i And the local density of the i-th data determines the outlier of the i-th data.

12. The device according to claim 11, wherein The outlier value of the i-th data is the distance D i and the ratio of the local density of the i-th data.

13. The device according to claim 11 or 12, wherein: The local density of the i-th data is determined by the distance between the i-th data and the j-th data in the m-th cluster, where 1≤j≤N and j≠i.

14. The device according to claim 13, wherein: The local density ρ of the i-th data i for where dist(x i , x j ) is the i-th data x i The jth data x j The distance between c is the preset cutoff distance.

15. An abnormal data detection device, comprising: Memory; A processor is coupled to the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 based on instructions stored in the memory.

16. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

17. A computer program comprising computer instructions, wherein when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Method and equipment for selecting cluster center of density peak clustering algorithm

    CN111582326A

  • Method, system and equipment for detecting impedance value of PCB (Printed Circuit Board) and storage medium

    CN115113022A

Cited By

  • Municipal underground pipeline inspection equipment and method

    CN120449059A

  • Remote meter reading method for concentrator

    CN120974218A

  • Abnormal alarm method and system for feed production control

    CN122387017A