Abnormal data detection method and device and storage medium
Patent Information
- Application Number
- CN202310869053.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-14
- Publication Date
- 2025-01-14
Smart Images
Figure CN119311672A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data mining, and in particular to an abnormal data detection method and device, and a storage medium. Background Art
[0002] As the business scale of telecom operators continues to grow, the traffic volume of their servers is growing exponentially. It is a very challenging task to select non-abnormal data from the numerous and complex data for data analysis. Therefore, it is necessary to extract features from telecom traffic data, compress data dimensions as much as possible, ensure correct classification, and detect outliers in the data.
[0003] The existing dynamic clustering algorithm does not need to set the number of categories in advance. It is a clustering algorithm that can automatically find the number of categories. It is described as follows:
[0004] Step 1: Build an undirected graph. For each node x i , assign the initial category class(x i )=i; Calculate the similarity between different nodes. If this value is greater than the preset threshold, build related edges and let the weight value be the value of the similarity.
[0005] Step 2: Traverse the process. Randomly select x i , if the nodes in the neighborhood belong to the same category, add up the weights of these nodes in sequence; select the largest category m as the current number of categories.
[0006] Step 3: Repeat step 2 until the traversal is completed and the number of classification clusters k is obtained. Summary of the invention
[0007] The inventors have noticed that in the related art, the dynamic clustering algorithm has the following deficiencies:
[0008] 1. In telecommunication traffic data, the weights of the edges in the network are unknown, so prior knowledge is required to obtain the weights of the edges. Since building a connection matrix with a small data set requires calculating the similarity between nodes, its time complexity is high and the network convergence speed is slow.
[0009] 2. The clustering results will be affected by the threshold value, and it is crucial to effectively select the appropriate threshold value.
[0010] Accordingly, the present disclosure provides an abnormal data detection solution that can effectively reduce computational complexity and accurately implement abnormal data detection.
[0011] In a first aspect of the present disclosure, a method for detecting abnormal data is provided, comprising: dividing a preset data set into multiple data subsets, wherein the data amount of each data subset is less than a preset data amount threshold; clustering each data subset to obtain multiple clustering results; clustering the cluster centers in the multiple clustering results so as to divide multiple data in the preset data set into multiple clusters; and in each of the multiple clusters, selecting abnormal data according to the distance between each data and the cluster center.
[0012] In some embodiments, in each of the plurality of clusters, selecting the abnormal data according to the distance between each data and the cluster center includes: calculating the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster. i , where 1≤m≤M, M is the total number of clusters, 1≤i≤N, N is the total number of data in the mth cluster; according to the distance D i Determine the outlier value of the i-th data; if the outlier value of the i-th data is greater than a preset outlier threshold, determine that the i-th data in the m-th cluster is outlier data.
[0013] In some embodiments, the outlier value of the i-th data is the distance D i .
[0014] In some embodiments, according to the distance D i Determining the outlier value of the i-th data includes: according to the distance D i And the local density of the i-th data determines the outlier of the i-th data.
[0015] In some embodiments, the outlier value of the i-th data is the distance D i and the ratio of the local density of the i-th data.
[0016] In some embodiments, the local density of the i-th data is determined by the distance between the i-th data and the j-th data in the m-th cluster, where 1≤j≤N and j≠i.
[0017] In some embodiments, the local density ρ of the i-th data i for
[0018]
[0019] where dist(x i , x j ) is the i-th data x i The jth data x j The distance between c is the preset cutoff distance.
[0020] In a second aspect of the present disclosure, an abnormal data detection device is provided, comprising: a first processing module, configured to divide a preset data set into multiple data subsets, wherein the data amount of each data subset is less than a preset data amount threshold; a second processing module, configured to cluster each of the data subsets to obtain multiple clustering results; a third processing module, configured to cluster the cluster centers in the multiple clustering results so as to divide the multiple data in the preset data set into multiple clusters; and a fourth processing module, configured to select abnormal data in each of the multiple clusters according to the distance between each data and the cluster center.
[0021] In some embodiments, the fourth processing module is configured to calculate the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster. i , where 1≤m≤M, M is the total number of clusters, 1≤i≤N, N is the total number of data in the mth cluster, according to the distance D i An outlier value of the i-th data is determined, and if the outlier value of the i-th data is greater than a preset outlier threshold, the i-th data in the m-th cluster is determined to be outlier data.
[0022] In some embodiments, the outlier value of the i-th data is the distance D i .
[0023] In some embodiments, the fourth processing module is configured to i And the local density of the i-th data determines the outlier of the i-th data.
[0024] In some embodiments, the outlier value of the i-th data is the distance D i and the ratio of the local density of the i-th data.
[0025] In some embodiments, the local density of the i-th data is determined by the distance between the i-th data and the j-th data in the m-th cluster, where 1≤j≤N and j≠i.
[0026] In some embodiments, the local density ρ of the i-th data i for
[0027]
[0028] where dist(x i , x j ) is the i-th data x i The jth data x j The distance between c is the preset cutoff distance.
[0029] In a third aspect of the present disclosure, an abnormal data detection device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute a method as described in any of the above embodiments based on instructions stored in the memory.
[0030] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method described in any of the above embodiments is implemented.
[0031] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0033] Figure 1 A flowchart of an abnormal data detection method according to an embodiment of the present disclosure;
[0034] Figure 2 This is a schematic diagram of the structure of an abnormal data detection device according to an embodiment of the present disclosure;
[0035] Figure 3 This is a schematic diagram of the structure of an abnormal data detection device according to another embodiment of the present disclosure;
[0036] Figure 4 Schematic diagram of time comparison of different clustering methods;
[0037] Figure 5(A)-Figure 5(D) Schematic diagram of AUC of different anomaly detection methods on 4 data sets;
[0038] Figure 6(A)-Figure 6(D) Schematic diagram of RMSE of different anomaly detection methods on 4 data sets. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0040] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure.
[0041] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0042] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.
[0043] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0044] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0045] Figure 1 The following is a flow chart of an abnormal data detection method according to an embodiment of the present disclosure. In some embodiments, the following abnormal data detection method is performed by an abnormal data detection device.
[0046] In step 101, a preset data set is divided into a plurality of data subsets, wherein the data volume of each data subset is less than a preset data volume threshold.
[0047] It should be noted here that by limiting the amount of data in each data subset, the performance of the clustering algorithm can be ensured while ensuring the stability of the clustering results of each data subset.
[0048] In step 102, each data subset is clustered to obtain a plurality of clustering results.
[0049] In step 103, cluster centers in the plurality of clustering results are clustered so as to divide the plurality of data in the preset data set into a plurality of clusters.
[0050] For example, the preset data set is divided into 2 data subsets. The clustering result of the first data subset has 2 clusters, and the centers of the 2 clusters are a1 and b1 respectively. The clustering result of the second data subset has 2 clusters, and the centers of the 2 clusters are a2 and c1 respectively. By clustering a1, b1, a2 and c1, three clusters are obtained, of which the first cluster includes a1 and a2, the second cluster includes b1, and the third cluster includes c1. In this case, the data originally included in the cluster with a1 as the center and the data with a2 as the center are marked as the first category, the data originally included in the cluster with b1 as the center is marked as the second category, and the data originally included in the cluster with c1 as the center is marked as the third category.
[0051] In step 104, in each of the plurality of clusters, abnormal data is selected according to the distance between each data and the cluster center.
[0052] In some embodiments, the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster is calculated. i , where 1≤m≤M, M is the total number of clusters, 1≤i≤N, N is the total number of data in the mth cluster, according to the distance D i The outlier value of the i-th data is determined. If the outlier value of the i-th data is greater than a preset outlier threshold, the i-th data in the m-th cluster is determined to be outlier data.
[0053] For example, the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster is i As shown in formula (1).
[0054] D i = dist(x im , C m )=‖x im -C m ‖ (1)
[0055] Among them, x im is the i-th data in the m-th cluster, C m is the cluster center of the mth cluster.
[0056] In some embodiments, the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster is directly calculated. i As the outlier of the i-th data. That is, the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster i The larger the value is, the greater the probability that the i-th data is an outlier.
[0057] In some embodiments, considering that there may be densely distributed data at the boundary between clusters, if only the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster is considered, iIn order to avoid misjudging non-abnormal data as abnormal data, according to the distance D i The local density of the ith data determines the outlier of the ith data.
[0058] In some embodiments, the outlier value of the i-th data is the distance D i The ratio of the local density of the ith data.
[0059] For example, the outlier score (x i ) is shown in formula (2).
[0060]
[0061] Among them, ρ i is the local density of the ith data.
[0062] From formula (2), we can see that the outlier value of the i-th data is related to dist(x im , C m ) is proportional to ρ i That is, data with smaller local density will be regarded as abnormal data with a high probability.
[0063] In some embodiments, the i-th data p i The local density of is determined by the distance between the i-th data and the j-th data in the m-th cluster, where 1≤j≤N and j≠i.
[0064] For example, the local density ρ of the i-th data i It is generated using the Gaussian kernel function, as shown in formula (3).
[0065]
[0066] Among them, dist(x i , x j ) is the i-th data x i The jth data x j The distance between c is the preset cutoff distance.
[0067] For example, the data in each cluster can be sorted according to a predetermined rule, and an appropriate value can be selected as the cutoff distance d according to the distance between adjacent data. c .
[0068] It should be noted here that the existing K-Means and K-Means++ clustering algorithms are random when selecting the initial cluster center, which leads to unstable clustering results, easy to fall into local optimal solutions, many iterations, and the number of clusters needs to be set manually. In the abnormal data detection method provided in the above embodiment of the present disclosure, the preset data set is divided into multiple data subsets, each data subset is clustered to obtain multiple clustering results, and the cluster centers in the multiple clustering results are clustered so that the multiple data in the preset data set are divided into multiple clusters, and then in each cluster, the abnormal data is selected according to the distance between each data and the cluster center. In other words, the present disclosure can effectively reduce the computational complexity by clustering each data subset instead of directly clustering the data set. At the same time, the present application does not need to manually set the number of clusters and can accurately realize abnormal data detection.
[0069] Figure 2 FIG. 1 is a schematic diagram of the structure of an abnormal data detection device according to an embodiment of the present disclosure. Figure 2 As shown, the abnormal data detection device includes a first processing module 21 , a second processing module 22 , a third processing module 23 and a fourth processing module 24 .
[0070] The first processing module 21 is configured to divide the preset data set into a plurality of data subsets, wherein the data volume of each data subset is less than a preset data volume threshold.
[0071] It should be noted here that by limiting the amount of data in each data subset, the performance of the clustering algorithm can be ensured while ensuring the stability of the clustering results of each data subset.
[0072] The second processing module 22 is configured to cluster each data subset to obtain a plurality of clustering results.
[0073] The third processing module 23 is configured to cluster the cluster centers in the plurality of clustering results so as to divide the plurality of data in the preset data set into a plurality of clusters.
[0074] The fourth processing module 24 is configured to select abnormal data in each of the plurality of clusters according to the distance between each data and the cluster center.
[0075] In some embodiments, the fourth processing module 24 is configured to calculate the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster. i , where 1≤m≤M, M is the total number of clusters, 1≤i≤N, N is the total number of data in the mth cluster, according to the distance D i The outlier value of the i-th data is determined. If the outlier value of the i-th data is greater than a preset outlier threshold, the i-th data in the m-th cluster is determined to be outlier data.
[0076] For example, the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster is i As shown in formula (1).
[0077] In some embodiments, the fourth processing module 24 directly calculates the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster. i As the outlier of the i-th data. That is, the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster i The larger the value is, the greater the probability that the i-th data is an outlier.
[0078] In some embodiments, the fourth processing module 24 is configured to i The local density of the ith data determines the outlier of the ith data.
[0079] In some embodiments, the outlier value of the i-th data is the distance D i The ratio of the local density of the ith data.
[0080] For example, the outlier score (x i ) is shown in formula (2).
[0081] In some embodiments, the local density of the i-th data is determined by the distance between the i-th data and the j-th data in the m-th cluster, where 1≤j≤N and j≠i.
[0082] For example, the local density ρ of the i-th data i It is generated using the Gaussian kernel function, as shown in formula (3).
[0083] Figure 3 FIG. 1 is a schematic diagram of the structure of an abnormal data detection device according to another embodiment of the present disclosure. Figure 3 As shown, the abnormal data detection device includes a memory 31 and a processor 32 .
[0084] The memory 31 is used to store instructions. The processor 32 is coupled to the memory 31. The processor 32 is configured to execute the instructions stored in the memory to implement the following. Figure 1 The method of any one of the embodiments.
[0085] like Figure 3 As shown, the abnormal data detection device also includes a communication interface 33 for information exchange with other devices. At the same time, the abnormal data detection device also includes a bus 34, through which the processor 32, the communication interface 33, and the memory 31 communicate with each other.
[0086] The memory 31 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory. The memory 31 may also be a memory array. The memory 31 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.
[0087] In addition, the processor 32 may be a central processing unit (CPU), or may be an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.
[0088] The present disclosure also relates to a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, which are executed by a processor to implement the following Figure 1 The method of any one of the embodiments.
[0089] The present disclosure is described below through specific examples.
[0090] Traffic data is one of the typical time series data, which is an important reference in system performance management, risk prevention and anomaly detection. The experimental data is the traffic data of a telecommunications branch. The traffic data from September 1 to October 15, 2022 was collected for experimental training, testing and verification. The data sampling interval is five minutes, and a total of 1,200 data were extracted. First, the data is preprocessed, cleaned and screened out 880 groups of valid data, 600 groups are used to complete the training of the model, and the remaining 280 groups are used to test the performance of the network model.
[0091] In order to evaluate the effectiveness of the anomaly detection algorithm proposed in this disclosure, three evaluation indicators are used: Average Precision AP (Average Precision), F1max and the closed area AUC (Area Under Curve) obtained by the observer operating characteristic curve. Assume that M h and M q They represent the real data and q candidate anomaly values respectively, and the precision value is expressed as the ratio of the real anomaly data to the candidate anomaly data:
[0092]
[0093] The average precision AP is expressed as the average of all values of P(q).
[0094] The recall rate is expressed as formula (5):
[0095]
[0096] P(q) represents the state of the real outlier among the candidate outliers, and R(q) is the state of the real outlier being successfully detected. F1(q) represents the harmonic mean of P(q) and R(q), which is defined as formula (6):
[0097]
[0098] F1max refers to the maximum value of F1(q) among q outliers.
[0099] The evaluation index disclosed in the present invention adopts AUC, which only requires calculating the area under the ROC (receiver operating characteristic curve), and its value is between [0, 1]. The larger the value, the better the abnormality detection effect.
[0100] In order to verify the time efficiency of the improved dynamic clustering algorithm proposed in this disclosure, the K-Means algorithm and the K-Means++ algorithm are used for clustering. The experimental results are detailed in Figure 4 .Depend on Figure 3 It can be seen that when the data scale increases, the clustering time of the K-Means algorithm and the K-Means++ algorithm shows a quadratic growth trend, and the clustering time of the present invention shows a linear growth trend. It can be seen that the solution provided by the present invention greatly improves the time efficiency, and the time complexity is reduced from the original O(n 2 ) becomes O(np).
[0101] In order to analyze the efficiency of the anomaly detection algorithm proposed in the present disclosure, the present disclosure uses the anomaly detection algorithm proposed in the present disclosure and other algorithms to perform performance testing and evaluation in four real data sets S1, S2, S3 and S4. These algorithms are: 3sigma, KNN (K Nearest Neighbors), SOS (Stochastic Outlier Selection) and PCA (Principal Component Analysis). The average of the three experimental results of 20min, 40min and 60min in the future is taken. The data is shown in Table 1. The experimental results are shown in Figure 5(A)-Figure 5(D) .
[0102]
[0103]
[0104] Table 1 Comparison of model prediction performance
[0105] It can be observed from Table 1 that, regardless of the training time of 20min, 40min or 60min, the AUC of the anomaly detection algorithm proposed in the present disclosure is higher than that of other anomaly detection algorithms. Taking the data of the S2 data set at 60min as an example, the algorithm proposed in the present disclosure is 26.3%, 22.9%, 35.8% and 2.2% higher than the AUC of 3sigma, KNN, SOS and PCA, respectively. It can be seen that the accuracy of the anomaly detection algorithm proposed in the present disclosure is generally good.
[0106] Figure 6(A)-Figure 6(D) The results show that the prediction performance of the model changes over time. As the prediction time becomes longer, the prediction error of the model will gradually increase, and the prediction effect will inevitably deteriorate. The RMSE (Root mean squared error) values of 3sigma, KNN, SOS and PCA continue to increase. Compared with these four models, the RMSE variation range of the model proposed in this disclosure is relatively small, indicating that the overall performance is better. Figure 6(A)-Figure 6(D) In the figure, label 1 represents the result of the 3sigma algorithm, label 2 represents the result of the KNN algorithm, label 3 represents the result of the SOS algorithm, label 4 represents the result of the PCA algorithm, and label 5 represents the result of the solution provided by the present disclosure.
[0107] In some embodiments, the functional unit described above can be implemented as a general-purpose processor, a programmable logic controller (PLC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component or any appropriate combination thereof for performing the functions described in the present disclosure.
[0108] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0109] The description of the present disclosure is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the present disclosure to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure, and to enable those of ordinary skill in the art to understand the present disclosure and thereby design various embodiments with various modifications suitable for specific uses.
Claims
1. A method for detecting abnormal data, comprising: Dividing a preset data set into multiple data subsets, wherein the data volume of each data subset is less than a preset data volume threshold; Clustering each of the data subsets to obtain multiple clustering results; Clustering the cluster centers in the plurality of clustering results so as to divide the plurality of data in the preset data set into a plurality of clusters; In each of the plurality of clusters, abnormal data is selected according to the distance between each data and the cluster center.
2. The method according to claim 1, wherein: In each of the plurality of clusters, selecting abnormal data according to the distance between each data and the cluster center includes: Calculate the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster i , where 1≤m≤M, M is the total number of clusters, 1≤i≤N, N is the total number of data in the mth cluster; According to the distance D i Determine the outlier value of the i-th data; If the abnormal value of the i-th data is greater than a preset abnormal value threshold, the i-th data in the m-th cluster is determined to be abnormal data.
3. The method according to claim 2, wherein: The outlier value of the i-th data is the distance D i .
4. The method according to claim 2, wherein: According to the distance D i Determining the outlier value of the i-th data includes: According to the distance D i And the local density of the i-th data determines the outlier of the i-th data.
5. The method according to claim 4, wherein: The outlier value of the i-th data is the distance D i and the ratio of the local density of the i-th data.
6. The method according to claim 4 or 5, wherein: The local density of the i-th data is determined by the distance between the i-th data and the j-th data in the m-th cluster, where 1≤j≤N and j≠i.
7. The method according to claim 6, wherein: The local density ρ of the i-th data i for where dist(x i , x j ) is the i-th data x i The jth data x j The distance between c is the preset cutoff distance.
8. An abnormal data detection device, comprising: A first processing module is configured to divide a preset data set into a plurality of data subsets, wherein the data volume of each data subset is less than a preset data volume threshold; A second processing module is configured to cluster each of the data subsets to obtain a plurality of clustering results; A third processing module is configured to cluster the cluster centers in the plurality of clustering results so as to divide the plurality of data in the preset data set into a plurality of clusters; The fourth processing module is configured to select abnormal data in each of the multiple clusters according to the distance between each data and the cluster center.
9. The device according to claim 8, wherein: The fourth processing module is configured to calculate the distance D between the i-th data in the m-th cluster and the cluster center of the m-th cluster. i , where 1≤m≤M, M is the total number of clusters, 1≤i≤N, N is the total number of data in the mth cluster, according to the distance D i An outlier value of the i-th data is determined, and if the outlier value of the i-th data is greater than a preset outlier threshold, the i-th data in the m-th cluster is determined to be outlier data.
10. The device according to claim 9, wherein: The outlier value of the i-th data is the distance D i .
11. The device according to claim 9, wherein: The fourth processing module is configured to i And the local density of the i-th data determines the outlier of the i-th data.
12. The device according to claim 11, wherein The outlier value of the i-th data is the distance D i and the ratio of the local density of the i-th data.
13. The device according to claim 11 or 12, wherein: The local density of the i-th data is determined by the distance between the i-th data and the j-th data in the m-th cluster, where 1≤j≤N and j≠i.
14. The device according to claim 13, wherein: The local density ρ of the i-th data i for where dist(x i , x j ) is the i-th data x i The jth data x j The distance between c is the preset cutoff distance.
15. An abnormal data detection device, comprising: Memory; A processor is coupled to the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 based on instructions stored in the memory.
16. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.