A data anomaly detection method, system and storage medium

Abstract

The application relates to the technical field of data mining, and discloses a data anomaly detection method, a system and a storage medium, the method adopts a self-adaptive clustering algorithm to divide data into multiple sub-clusters, calculates the connection strength between adjacent sub-clusters, and then uses an improved Floyd algorithm to obtain a connection strength matrix between any sub-clusters, the values of the matrix can reasonably represent the connection degree between any two sub-clusters, the sub-cluster center position and the connection strength matrix can reflect the distribution of data, the similarity between each sample and the sub-cluster calculated based on the similarity strategy is adjusted by using the connection strength matrix, and then the anomaly score of each sample is calculated according to the similarity; the method for determining abnormal data by calculating the anomaly score considers global and local information of a data set, can detect not only scattered point anomalies in sparse areas but also abnormal data gathered into clusters, reduces the influence of abnormal data distribution on the detection result, and improves the detection accuracy of abnormal data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data mining, in particular to a data anomaly detection method and system and a storage medium. BACKGROUND

[0002] In the field of data anomaly detection, data that is greatly different from most data behaviors is abnormal data, which is sometimes also referred to as outliers, noise, novelty, etc. At present, the application range of data anomaly detection is very wide, and common application fields include fraud detection, network intrusion detection, disease detection, industrial hazard detection and data cleaning. The existing data anomaly detection method mainly includes an anomaly detection method based on similarity / proximity. This method mainly uses the similarity / proximity between sample points in distance or density as the basis for judging anomalies. The sample points that are different from most sample points in density or distance are highly likely to be abnormal. This type of method generally assumes that abnormal samples are distributed in areas with low density or in remote edges. The abnormal samples detected by this type of method have high interpretability, but the detected abnormal samples are usually scattered abnormal samples, and cannot detect abnormal samples that are clustered into clusters.

[0003] The prior art discloses a heterogeneous multi-source time sequence data anomaly recognition method for a compression molding device, which comprises: a sensor data set construction module, which generates a sensor data vector according to the correspondence between original data and collected time, and performs standardization processing, and then constructs a sensor data set based on the standardized data vector; a clustering cluster and intra-cluster data detection set construction module, which constructs a clustering cluster in the sensor data set through a K-means algorithm, and constructs an intra-cluster data detection set corresponding to the clustering cluster; and an abnormal data recognition module, which puts the data in each clustering cluster into the intra-cluster data detection set for anomaly recognition through a LOF algorithm, and performs corresponding marking. The prior art uses the K-means algorithm to construct the clustering cluster in the sensor data set, and uses the LOF algorithm to detect the data detection set. This data anomaly detection method is greatly affected by data shape and distribution, and the detection accuracy is not high when the abnormal data is unevenly distributed, such as when the abnormal data is clustered. SUMMARY

[0004] The purpose of the present application is to provide an abnormal data detection method, system and storage medium based on an adaptive data division method and a data weighted undirected graph representation, which can improve the detection accuracy when abnormal data is clustered.

[0005] In order to achieve the above-mentioned purpose, the present application provides a data anomaly detection method, comprising the following steps:

[0006] Step S1: obtaining a data set to be detected;

[0007] Step S2: Divide the dataset to be detected into K subclusters using an adaptive partitioning method, and retain the cluster center position of each subcluster, where each of the K subclusters contains multiple sample points;

[0008] Step S3: Input all adjacent subclusters and the cluster center positions of the corresponding subclusters in the K subclusters into the algorithm based on the Gaussian mixture model, and output the connection strength between all adjacent subclusters. The specific process is as follows:

[0009] Step S3.1: All the K subclusters with adjacent relationships are All sample points in the cluster are projected to the center prototypes of two adjacent sub-clusters. On the one-dimensional line where it is located, the one-dimensional subcluster after dimensionality reduction projection can be obtained ;

[0010] Step S3.2: One-dimensional sub-cluster after dimensionality reduction projection Calculate the one-dimensional subcluster after the dimensionality reduction projection The corresponding one-dimensional mean and the corresponding one-dimensional variance According to the one-dimensional mean and the corresponding one-dimensional variance Define the sub-cluster The corresponding one-dimensional Gaussian probability density function and

[0011] Step S3.3: All the K subclusters with adjacent relationships are Add up all the sample points in the to get the total number of sample points , and then calculate all subclusters with adjacent relationships The number of sample points within the total sample points The proportion of all subclusters with adjacent relationships is obtained Corresponding ratio ;

[0012] Step S3.4: The ratio Multiply them with the corresponding one-dimensional Gaussian probability density function respectively, and then add the multiplication results to obtain a one-dimensional two-variable Gaussian mixture probability density function The one-dimensional two-variable Gaussian mixture probability density function F is a bimodal distribution, and the minimum probability density value between the two peaks is used as the subclusters with any two adjacent relationships. The initial connection strength between

[0013] Step S3.5: repeating step S3.1 to step S3.4 to obtain initial connection strengths between all existing adjacent sub-clusters in the K sub-clusters, normalizing the initial connection strengths to obtain connection strengths between all existing adjacent sub-clusters in the K sub-clusters;

[0014] Step S4: constructing a weighted undirected graph between adjacent sub-clusters according to the adjacent relationship between sub-clusters and the connection strength between adjacent sub-clusters;

[0015] Step S5: obtaining a K*K adjacency matrix W0 from the weighted undirected graph between adjacent sub-clusters, inputting the K*K adjacency matrix W0 into the improved Floyd algorithm to obtain a connection strength matrix W between any sub-clusters, the connection strength matrix W between any sub-clusters being represented as an adjacency matrix of the weighted undirected graph between any sub-clusters in the K sub-clusters;

[0016] Step S6: calculating similarity between each sample point and the cluster centers of the K sub-clusters using a similarity strategy to obtain initial similarity, adjusting the initial similarity by the connection strength matrix W between any two sub-clusters to obtain adjusted similarity, calculating an anomaly score according to the adjusted similarity, ranking the anomaly scores in descending order, and the sample points corresponding to the top anomaly scores being abnormal data.

[0017] Further, the specific implementation process of step S2 includes:

[0018] Step S2.1: assuming a to-be-detected data set , wherein x i represents the i-th to-be-detected sample point in the to-be-detected data set, i has a value of 1 to n, n represents that the to-be-detected data set has n sample points, and K0=n / 10 sample points are randomly selected from the to-be-detected data set X as an initial sub-cluster cluster center set , wherein represents the j-th sub-cluster cluster center in the initial sub-cluster cluster center set, and j has a value of 1 to K0;

[0019] Step S2.2: using a rival penalty competitive learning formula to reward and punish each sub-cluster cluster center in the initial sub-cluster cluster center set Seed for each sample point in the to-be-detected data set, defining a winner number set in the whole rewarding and punishing process, wherein is one-to-one corresponding to each sub-cluster cluster center , updating and recording the number of times of each sub-cluster cluster center as a winner, and when each sample point in the to-be-detected data set participates in rewarding and punishing the initial sub-cluster cluster center Rewards and punishments, complete a round of iteration, after each round of iteration, the set The number of winners is less than 2. Delete from the initial sub-cluster cluster center set Seed, the initial sub-cluster cluster center set After deletion, there are K sub-cluster centers remaining, K≤K0=n / 10, and the sub-cluster centers also become Repeat this step until the number of subclusters and the position of subclusters in the Seed set converge to obtain the final

[0020] Step S2.3: Set the sub-cluster centers after convergence Each prototype in As the sub-cluster center, according to the principle of the closest distance, each sample point Belong to the sub-cluster center closest to it , the sample points As the sub-cluster core The cluster members are divided into K subclusters, and the K subclusters are recorded as C1, C2, ..., C K .

[0021] Furthermore, the opponent penalty competitive learning formula in step S2.2 is determined as follows:

[0022]

[0023] Among them, x i represents the current sample point, Represents the current sample point x i The closest sub-cluster center, Indicates that the current sample point x is found by the first formula above i The index of the nearest sub-cluster center, The sub-cluster center representing the current reward or punishment Indicates the number of current sub-cluster centers, 0 represents the number of initial sub-cluster centers, represents the penalty coefficient, Expressing defeat to the opponent the intensity of punishment, It means that as the number of sub-cluster centers decreases, the rewards and punishments for each cluster center should be reduced. t represents the number of rewards and punishments. Represents the sub-cluster center The updated value after receiving the reward or penalty from the sample point in the dataset X to be tested for the tth time, Indicates the The next update after being rewarded or penalized.

[0024] Furthermore, the learning rate Penalty coefficient The value range is .

[0025] Furthermore, the learning rate Penalty coefficient The value of .

[0026] Furthermore, the specific process of obtaining the connection strength matrix W between any two subclusters in step S5 is as follows:

[0027] The improved Floyd algorithm updates and improves a weighted undirected graph by continuously adding subsequent nodes, so that the weight between any two nodes in the weighted undirected graph is maximized. The specific steps include:

[0028] Step S5.1: The K×K adjacency matrix W0 is a symmetric matrix, let D0=W0, d0( ) represents the first Nodes and The weight between nodes is the size of the weight, the node is the sub-cluster, and the weight is the size of the The subcluster represented by the node and the The connection strength between the subclusters represented by the nodes;

[0029] Step S5.2: Let k = 1, where k represents taking the kth node in the D0 matrix as a subsequent node and using the subsequent node to expand the path, wherein the value of k ranges from 1 to K;

[0030] Step S5.3: D k-1 Represents the matrix after the last update, if D k-1 Each element d in the matrix k-1 ( ) satisfies d k-1 ( ) d k-1 ( )>d k-1 ( ) is updated The first Nodes and The weight between nodes is d k ( )=d k-1 ( ) d k-1 ( );If D k-1 Each element d in the matrixk-1 ( ) does not satisfy d k-1 ( ) d k-1 ( )>d k-1 ( ), then the The first Nodes and The weight between nodes is d k ( )=d k-1 ( ); until all the For each element in the matrix, get the matrix ;

[0031] Step S5.4: If k <K,则令k=k+1,重复步骤S5.3,直到k=K停止,此时 The matrix is ​​the connection strength matrix between any two subclusters, that is, =W.

[0032] Furthermore, the specific process of step S6 is as follows:

[0033] Step S6.1: Calculate each sample point in the data set to be tested and the converged sub-cluster center set Each sub-cluster center in The similarity calculation formula of similarity is:

[0034]

[0035] in, express The standard deviation of the sub-cluster represented by xi can be obtained according to the calculated similarity to obtain an n×K similarity matrix S, where n represents the number of sample points in the data set to be tested, K represents the K sub-clusters into which the data set to be tested is divided, and S(i, j) = Similarity(xi, mj) represents the similarity between the i-th sample point and the j-th sub-cluster center.

[0036] Step S6.2: Based on each sample point The relationship between the sample points and the K subclusters is defined as an n×K matrix C, where n represents the number of sample points in the dataset to be tested, and K represents the K subclusters into which the dataset to be tested is divided. belong The subcluster represented by , then C ( ) = 1, if the sample point Not The subcluster represented by , then C ( ) = 0, where j ranges from 1 to K;

[0037] Step S6.3: Based on each sample point The similarity between each sample point and the center of other sub-clusters is adjusted by the connection strength between the sub-cluster to which it belongs and other sub-clusters, and each sample point in the data set to be tested is obtained. The adjusted n×K similarity matrix to each sub-cluster center prototype , sample points The connection strength between the sub-cluster to which it belongs and other sub-clusters is obtained according to the connection strength matrix W between any two sub-clusters. The adjusted n×K similarity matrix Determined by the following formula:

[0038]

[0039] in, Indicates the similarity between the i-th sample point in the data set to be detected and the j-th sub-cluster center, Represents the connection strength weight between the kth sub-cluster and the jth sub-cluster, k is the sample point x i The subcluster to which it belongs can be obtained through the matrix C, that is, k satisfies C(i, k) = 1;

[0040] Step S6.4: The similarity between each sample point in the dataset to be detected and the center of each sub-cluster can be obtained based on the adjusted n×K similarity matrix Sim, and the anomaly score is calculated based on the similarity. The specific process is as follows:

[0041] Add each row in the adjusted n×K similarity matrix Sim to obtain an n×1 row vector; then normalize the row vector, subtract the normalized row vector from 1, and obtain the anomaly score corresponding to each sample point in the sample to be detected. Rank the anomaly scores in descending order, and the sample points corresponding to the top anomaly scores are the abnormal data.

[0042] In addition, the present invention also provides a data anomaly detection system, comprising:

[0043] Acquisition module: used to obtain the data set to be tested;

[0044] Partitioning module: used to divide the data set to be detected into K subclusters using an adaptive partitioning method, and retain the cluster center position of each subcluster, each of the K subclusters contains multiple sample points;

[0045] A module for calculating the connection strength between adjacent sub-clusters: used for inputting all existing adjacent sub-clusters and the cluster center positions of the corresponding sub-clusters in K sub-clusters into a Gaussian mixture model-based algorithm, and outputting the connection strength between all existing adjacent sub-clusters.

[0046] A module for obtaining a weighted undirected graph: used for constructing a weighted undirected graph between adjacent sub-clusters according to the adjacent relationship between sub-clusters and the connection strength between adjacent sub-clusters.

[0047] A module for obtaining a connection strength matrix between arbitrary sub-clusters: used for obtaining a K*K adjacency matrix W0 from the weighted undirected graph between adjacent sub-clusters, and inputting the K*K adjacency matrix W0 into an improved Floyd algorithm to obtain a connection strength matrix W between arbitrary sub-clusters, wherein the connection strength matrix W between arbitrary sub-clusters represents the adjacency matrix of the weighted undirected graph between arbitrary sub-clusters in the K sub-clusters.

[0048] A detection module: used for calculating the similarity between each sample point and the cluster center of the K sub-clusters using a similarity strategy to obtain an initial similarity, adjusting the initial similarity by the connection strength matrix W between arbitrary sub-clusters to obtain an adjusted similarity, calculating an anomaly score according to the adjusted similarity, ranking the anomaly scores in descending order, and regarding the sample point corresponding to the anomaly score at the top of the ranking as an abnormal data.

[0049] In addition, the application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the data anomaly detection method.

[0050] The application divides data into multiple sub-clusters by using an adaptive clustering algorithm, calculates the connection strength between adjacent sub-clusters, and obtains a connection strength matrix between arbitrary sub-clusters by using an improved Floyd algorithm, wherein each value in the matrix can reasonably represent the connection degree between any two sub-clusters, the cluster center position of the sub-clusters and the connection strength matrix can reflect the distribution of data, the similarity between each sample and the sub-clusters calculated based on the similarity strategy is adjusted by using the connection strength matrix, and the anomaly score of each sample is calculated according to the similarity, wherein the method for determining abnormal data by calculating the anomaly score considers the global and local information of the data set, can detect not only the scattered point anomaly in the sparse area but also the abnormal data gathered into clusters, reduces the influence of the abnormal data distributed in clusters on the detection result, and improves the detection accuracy of abnormal data. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of a data anomaly detection method according to an embodiment of the application;

[0052] Figure 2is a structural block diagram of a data anomaly detection system according to an embodiment of the present application;

[0053] Figure 3 is a flow chart of a data set adaptive partitioning sub-cluster method according to an embodiment of the present application;

[0054] Figure 4 is a flow chart of data anomaly detection based on a connection strength matrix according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] The specific embodiments of the present application will be further described in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present application, but not to limit the scope of the present application.

[0056] Embodiment one:

[0057] Please refer to Figure 1 A data anomaly detection method according to a preferred embodiment of the present application includes the following steps:

[0058] Step S1: obtaining a data set to be detected;

[0059] Step S2: dividing the data set to be detected into K sub-clusters using an adaptive partitioning method, and retaining the cluster center position of each sub-cluster, each of the K sub-clusters containing a plurality of sample points;

[0060] Step S3: inputting all pairs of sub-clusters having an adjacent relationship in the K sub-clusters and the cluster center positions of the corresponding sub-clusters into a Gaussian mixture model-based algorithm, and outputting the connection strength between all pairs of sub-clusters having an adjacent relationship;

[0061] Step S4: constructing a weighted undirected graph between adjacent sub-clusters according to the adjacent relationship between sub-clusters and the connection strength between adjacent sub-clusters;

[0062] Step S5: obtaining a K*K adjacency matrix W0 from the weighted undirected graph between adjacent sub-clusters, and inputting the K*K adjacency matrix W0 into an improved Floyd algorithm to obtain a connection strength matrix W between any sub-clusters, the connection strength matrix W between any sub-clusters being represented as an adjacency matrix of the weighted undirected graph between any sub-clusters in the K sub-clusters;

[0063] Step S6: calculating the similarity between each sample point and the cluster center of the K sub-clusters using a similarity strategy to obtain an initial similarity, adjusting the initial similarity using the connection strength matrix W between any two sub-clusters to obtain an adjusted similarity, calculating an anomaly score according to the adjusted similarity, ranking the anomaly scores in descending order, and the sample points corresponding to the top-ranked anomaly scores being abnormal data.

[0064] The application divides data into multiple sub-clusters by using an adaptive clustering algorithm, calculates the connection strength between adjacent sub-clusters, and then uses an improved Floyd algorithm to obtain the connection strength matrix between any sub-clusters, so that each value in the matrix can reasonably represent the connection degree between any two sub-clusters. The cluster center position of the sub-cluster and the connection strength matrix can reflect the distribution of data. Then, the similarity between each sample and the sub-cluster calculated based on the similarity strategy is adjusted by using the connection strength matrix, and the abnormal score of each sample is calculated according to the similarity. This method of determining abnormal data by calculating the abnormal score considers the global and local information of the data set, can not only detect scattered point anomalies in sparse areas, but also detect abnormal data aggregated into clusters, reduces the influence of the distribution of abnormal data on the detection result, and improves the detection accuracy of abnormal data.

[0065] Embodiment two

[0066] Please refer to Figure 1 The data anomaly detection method of the embodiment of the application comprises the following steps:

[0067] Step S1: obtaining a data set to be detected;

[0068] In this embodiment, the data set to be detected should include multiple sample points, and the abnormal data in the multiple sample points can be dispersed or concentrated.

[0069] Step S2: dividing the data set to be detected into K sub-clusters by using an adaptive division method, and retaining the cluster center position of each sub-cluster, each of the K sub-clusters containing multiple sample points;

[0070] In this embodiment, please refer to Figure 3 The specific process of dividing the data set to be detected into K sub-clusters by using an adaptive division method comprises:

[0071] Step S2.1: assuming that the data set to be detected is , wherein x i represents the i-th sample point to be detected in the data set to be detected, i takes a value from 1 to n, n represents that the data set to be detected has n sample points, K0=n / 10 sample points are randomly selected from the data set to be detected X as an initial sub-cluster cluster center set Seed , wherein represents the j-th sub-cluster cluster center in the initial sub-cluster cluster center set Seed, j takes a value from 1 to K0;

[0072] Step S2.2: rewarding and punishing each sub-cluster cluster center in the initial sub-cluster cluster center set Seed by using a formula of a game against a rival for each sample point in the data set to be detected , wherein , wherein With each of the sub-cluster heart One-to-one correspondence, update and record each sub-cluster center Number of times as a winner , when each sample point in the dataset to be detected participates in the initial sub-cluster centroid Rewards and punishments, complete a round of iteration, after each round of iteration, the set The number of winners is less than 2. Delete from the initial sub-cluster cluster center set Seed, the initial sub-cluster cluster center set After deletion, there are K sub-cluster centers remaining, K≤K0=n / 10, and the sub-cluster centers also become Repeat this step until the number of subclusters and the position of subclusters in the Seed set converge to obtain the final

[0073] Step S2.3: Set the sub-cluster centers after convergence Each prototype in As the sub-cluster center, according to the principle of the closest distance, each sample point Belong to the sub-cluster center closest to it , the sample points As the sub-cluster core The cluster members are divided into K subclusters, and the K subclusters are recorded as C1, C2, ..., C K .

[0074] It should be noted that, in this embodiment, the opponent penalty competitive learning formula in step S2.2 is determined as follows:

[0075] Among them, x i represents the current sample point, Represents the current sample point x i The closest sub-cluster center, Indicates that the current sample point x is found by the first formula above i The index of the nearest sub-cluster center, The sub-cluster center representing the current reward or punishment Indicates the number of current sub-cluster centers, 0 represents the number of initial sub-cluster centers, represents the penalty coefficient, Expressing defeat to the opponent the intensity of punishment, It means that as the number of sub-cluster centers decreases, the rewards and punishments for each cluster center should be reduced. t represents the number of rewards and punishments. Represents the sub-cluster center the value updated after the t-th time of receiving the reward or penalty from the sample point in the to-be-detected data set X, representing the next update after the reward or penalty.

[0076] It should be noted that, in the embodiment, the learning rate the value range of the penalty coefficient . .

[0077] It should be noted that, in the embodiment, the learning rate the value of the penalty coefficient . .

[0078] Step S3: input the K sub-clusters and the cluster center positions of the sub-clusters into a Gaussian mixture model-based algorithm, and output the connection strength between any two sub-clusters having a neighboring relationship;

[0079] In the embodiment, the specific implementation process of step S3 is as follows:

[0080] Step S3.1: project all sample points in all sub-clusters having a neighboring relationship in the K sub-clusters into a one-dimensional line where the two neighboring sub-cluster cluster centers are located, to obtain one-dimensional sub-clusters after dimension reduction projection. Step S3.2: calculate the one-dimensional mean and the one-dimensional variance corresponding to the one-dimensional sub-clusters after dimension reduction projection.

[0081] Step S3.3: add all sample points in all sub-clusters having a neighboring relationship in the K sub-clusters to obtain the total number of sample points , and then calculate the proportion of the number of sample points in all sub-clusters having a neighboring relationship in the total number of sample points , to obtain the proportion corresponding to all sub-clusters having a neighboring relationship. Step S3.4: define the one-dimensional Gaussian probability density function and

[0082] Step S3.3: add all sample points in all sub-clusters having a neighboring relationship in the K sub-clusters to obtain the total number of sample points , and then calculate the proportion of the number of sample points in all sub-clusters having a neighboring relationship in the total number of sample points , to obtain the proportion corresponding to all sub-clusters having a neighboring relationship. ; ​​​​​

[0083] Step S3.4: The ratio Multiply them with the corresponding one-dimensional Gaussian probability density function respectively, and then add the multiplication results to obtain a one-dimensional two-variable Gaussian mixture probability density function The one-dimensional two-variable Gaussian mixture probability density function F is a bimodal distribution, and the minimum probability density value between the two peaks is used as the subclusters with any two adjacent relationships. The initial connection strength between

[0084] Step S3.5: Repeat steps S3.1 to S3.4 to obtain the initial connection strengths between all adjacent subclusters in the K subclusters, normalize the initial connection strengths, and obtain the connection strengths between all adjacent subclusters in the K subclusters.

[0085] Step S4: construct a weighted undirected graph between adjacent subclusters based on the adjacent relationship between subclusters and the connection strength between adjacent subclusters;

[0086] In this embodiment, each node in a weighted undirected graph corresponds to a subcluster. Two adjacent subclusters have an undirected edge between them, and the weight of the edge is set to the normalized connection strength between the two subclusters corresponding to the two nodes. In the adjacency matrix representation of this weighted undirected graph, the weights in the matrix are set as follows: the weight between adjacent subclusters is the normalized connection strength, the weight between a subcluster and itself is 1, and the weight between a subcluster and a non-adjacent subcluster is 0.

[0087] Step S5: A K*K adjacency matrix W0 can be obtained based on the weighted undirected graph between adjacent subclusters. The K*K adjacency matrix W0 is input into the improved Floyd algorithm to obtain the connection strength matrix W between any subclusters. The connection strength matrix W between any subclusters is expressed as the adjacency matrix of the weighted undirected graph between any subclusters in the K subclusters.

[0088] The improved Floyd algorithm updates and improves the weighted undirected graph by continuously adding subsequent nodes, so that the weight between any two nodes in the weighted undirected graph is maximized. The specific steps include:

[0089] Step S5.1: The K×K adjacency matrix W0 is a symmetric matrix, let D0=W0, d0( ) represents the first Nodes and The weight between nodes is the size of the weight, the node is the sub-cluster, and the weight is the size of the The subcluster represented by the node and the The connection strength between the subclusters represented by the nodes;

[0090] Step S5.2: Let k = 1, where k represents taking the kth node in the D0 matrix as a subsequent node and using the subsequent node to expand the path, wherein the value of k ranges from 1 to K;

[0091] Step S5.3: D k-1 Represents the matrix after the last update, if D k-1 Each element d in the matrix k-1 ( ) satisfies d k-1 ( ) d k-1 ( )>d k-1 ( ) is updated The first Nodes and The weight between nodes is d k ( )=d k-1 ( ) d k-1 ( );If D k-1 Each element d in the matrix k-1 ( ) does not satisfy d k-1 ( ) d k-1 ( )>d k-1 ( ), then the The first Nodes and The weight between nodes is d k ( )=d k-1 ( ); until all the For each element in the matrix, get the matrix ;

[0092] Step S5.4: If k <K,则令k=k+1,重复步骤S5.3,直到k=K停止,此时 The matrix is ​​the connection strength matrix between any two subclusters, that is, =W.

[0093] Step S6: Use the similarity strategy to calculate the similarity between each sample point and the K sub-cluster centers to obtain the initial similarity. Adjust the initial similarity by the connection strength matrix W between any two sub-clusters to obtain the adjusted similarity. Calculate the anomaly score based on the adjusted similarity and rank the anomaly scores in descending order. The sample points corresponding to the top anomaly scores are the abnormal data.

[0094] In this embodiment, the specific implementation process of step S6 is:

[0095] Step S6.1: Calculate each sample point in the data set to be tested and the converged sub-cluster center set Each sub-cluster center in The similarity calculation formula of similarity is:

[0096]

[0097] in, express The standard deviation of the sub-cluster represented by xi can be obtained according to the calculated similarity to obtain an n×K similarity matrix S, where n represents the number of sample points in the data set to be tested, K represents the K sub-clusters into which the data set to be tested is divided, and S(i, j) = Similarity(xi, mj) represents the similarity between the i-th sample point and the j-th sub-cluster center.

[0098] Step S6.2: Based on each sample point The relationship between the sample points and the K subclusters is defined as an n×K matrix C, where n represents the number of sample points in the dataset to be tested, and K represents the K subclusters into which the dataset to be tested is divided. belong The subcluster represented by , then C ( ) = 1, if the sample point Not The subcluster represented by , then C ( ) = 0, where j ranges from 1 to K;

[0099] Step S6.3: Based on each sample point The similarity between each sample point and the center of other sub-clusters is adjusted by the connection strength between the sub-cluster to which it belongs and other sub-clusters, and each sample point in the data set to be tested is obtained. The adjusted n×K similarity matrix to each sub-cluster center prototype , sample points The connection strength between the sub-cluster to which it belongs and other sub-clusters is obtained according to the connection strength matrix W between any two sub-clusters. The adjusted n×K similarity matrix is determined by the following formula:

[0100]

[0101] wherein, represents the similarity between the i-th sample point in the to-be-detected data set and the cluster center of the j-th sub-cluster, represents the connection strength weight between the k-th sub-cluster and the j-th sub-cluster, k is the sample point x i The sub-cluster to which the sample point belongs can be obtained by a matrix C, that is, k satisfies C(i, k) = 1.

[0102] Step S6.4: According to the adjusted n x K similarity matrix Sim, the similarity between each sample point in the to-be-detected data set and each sub-cluster cluster center can be obtained, and an anomaly score is calculated according to the similarity, and the specific process is as follows:

[0103] Add each row in the adjusted n x K similarity matrix Sim to obtain an n x 1 row vector, then normalize the row vector, subtract the normalized row vector from 1 to obtain an anomaly score corresponding to each sample point in the to-be-detected sample, and rank the anomaly scores in descending order, and the sample point corresponding to the anomaly score at the top of the ranking is the abnormal data.

[0104] Embodiment three:

[0105] Please refer to Figure 2 The present application provides a data anomaly detection system, comprising:

[0106] An acquisition module is configured to acquire a to-be-detected data set;

[0107] A division module is configured to divide the to-be-detected data set into K sub-clusters using an adaptive division method, and to retain the cluster center position of each sub-cluster, each of the K sub-clusters containing a plurality of sample points;

[0108] A connection strength calculation module is configured to input the cluster center positions of all pairs of sub-clusters having an adjacent relationship in the K sub-clusters and the corresponding sub-clusters into a Gaussian mixture model-based algorithm, and to output the connection strength between all pairs of sub-clusters having an adjacent relationship;

[0109] A weighted undirected graph acquisition module is configured to construct a weighted undirected graph between adjacent sub-clusters according to the adjacent relationship between the sub-clusters and the connection strength between the adjacent sub-clusters.

[0110] An arbitrary sub-cluster connection strength matrix obtaining module: a K*K adjacency matrix W0 of a weighted undirected graph between adjacent sub-clusters is obtained, the K*K adjacency matrix W0 is input into an improved Floyd algorithm, and a connection strength matrix W between arbitrary sub-clusters is obtained, wherein the connection strength matrix W between arbitrary sub-clusters is represented as an adjacency matrix of a weighted undirected graph between arbitrary sub-clusters in the K sub-clusters.

[0111] A detection module: similarity between each sample point and a cluster center of the K sub-clusters is calculated using a similarity strategy, an initial similarity is obtained, the connection strength matrix W between arbitrary two sub-clusters is adjusted on the initial similarity, an adjusted similarity is obtained, an anomaly score is calculated according to the adjusted similarity, the anomaly score is ranked in descending order, and a sample point corresponding to a top anomaly score is abnormal data.

[0112] Embodiment four:

[0113] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program of a data anomaly detection method, and a computer storage program of a data anomaly detection method is executed by a processor to realize steps of a data anomaly detection method.

[0114] In summary, the present application divides data into multiple sub-clusters by using an adaptive clustering algorithm, calculates connection strength between adjacent sub-clusters, and then uses an improved Floyd algorithm to obtain a connection strength matrix between arbitrary sub-clusters, wherein each value of the matrix can reasonably represent connection degree between two sub-clusters, and the cluster center position of the sub-clusters and the connection strength matrix can reflect data distribution. Then, the connection strength matrix is used to adjust similarity between each sample and a sub-cluster calculated based on a similarity strategy, and then an anomaly score of each sample is calculated according to the similarity. This method of determining abnormal data by calculating an anomaly score considers global and local information of a data set, can detect not only sparse area point anomalies but also clustered abnormal data, reduces the influence of concentrated abnormal data distribution on detection results, and improves detection accuracy of abnormal data.

[0115] The above only describes preferred embodiments of the present application, and it should be noted that, for those skilled in the art, without departing from the technical principles of the present application, several improvements and replacements can be made, and these improvements and replacements should also be considered as the protection scope of the present application.

Claims

1. A data anomaly detection method, characterized in that: The following steps are involved: Step S1: Obtain the data set to be tested; Step S2: Divide the dataset to be detected into K subclusters using an adaptive partitioning method, and retain the cluster center position of each subcluster, where each of the K subclusters contains multiple sample points; Step S3: Input all adjacent subclusters and the cluster center positions of the corresponding subclusters in the K subclusters into the algorithm based on the Gaussian mixture model, and output the connection strength between all adjacent subclusters. The specific process is as follows: Step S3.1: All the K subclusters with adjacent relationships are All sample points in the cluster are projected to the center prototypes of two adjacent sub-clusters. On the one-dimensional line where it is located, the one-dimensional subcluster after dimensionality reduction projection can be obtained ; Step S3.2: One-dimensional sub-cluster after dimensionality reduction projection Calculate the one-dimensional subcluster after the dimensionality reduction projection The corresponding one-dimensional mean and the corresponding one-dimensional variance According to the one-dimensional mean and the corresponding one-dimensional variance Define the sub-cluster The corresponding one-dimensional Gaussian probability density function and Step S3.3: All the K subclusters with adjacent relationships are Add up all the sample points in the to get the total number of sample points , and then calculate all subclusters with adjacent relationships The number of sample points within the total sample points The proportion of all subclusters with adjacent relationships is obtained Corresponding ratio , ; Step S3.4: The ratio , Multiply them with the corresponding one-dimensional Gaussian probability density function respectively, and then add the multiplication results to obtain a one-dimensional two-variable Gaussian mixture probability density function The one-dimensional two-variable Gaussian mixture probability density function F is a bimodal distribution, and the minimum probability density value between the two peaks is used as the value of any two adjacent subclusters. The initial connection strength between Step S3.5: Repeat steps S3.1 to S3.4 to obtain the initial connection strengths between all adjacent subclusters in the K subclusters, normalize the initial connection strengths, and obtain the connection strengths between all adjacent subclusters in the K subclusters; Step S4: construct a weighted undirected graph between adjacent subclusters based on the adjacent relationship between subclusters and the connection strength between adjacent subclusters; Step S5: A K*K adjacency matrix W0 can be obtained based on the weighted undirected graph between the adjacent subclusters. The K*K adjacency matrix W0 is input into the improved Floyd algorithm to obtain a connection strength matrix W between any subclusters. The connection strength matrix W between any subclusters is expressed as the adjacency matrix of the weighted undirected graph between any subclusters in the K subclusters. Step S6: Use the similarity strategy to calculate the similarity between each sample point and the K sub-cluster centers to obtain the initial similarity, adjust the connection strength matrix W between any two sub-clusters by the initial similarity to obtain the adjusted similarity, calculate the anomaly score based on the adjusted similarity, and rank the anomaly scores in descending order. The sample points corresponding to the top anomaly scores are the abnormal data.

2. A data anomaly detection method according to claim 1, characterized in that: The specific implementation process of step S2 includes: Step S2.1: Assume the dataset to be tested , where x i Represents the i-th sample point to be detected in the data set to be detected, the value of i ranges from 1 to n, n represents that the data set to be detected has n sample points, and K0=n / 10 sample points are randomly selected from the data set to be detected X as the initial sub-cluster cluster center set ,in represents the jth sub-cluster center in the initial sub-cluster center set, where the value of j ranges from 1 to K0; Step S2.2: Use the opponent penalty competitive learning formula to learn each sample point in the dataset to be tested for each sub-cluster center in the initial sub-cluster center set Seed. Carry out rewards and punishments, and define the number of winners in the entire reward and punishment process ,in With each of the sub-cluster heart One-to-one correspondence, update and record each sub-cluster center Number of times as a winner , when each sample point in the dataset to be detected participates in the initial sub-cluster centroid Rewards and punishments, complete a round of iteration, after each round of iteration, the set The number of winners is less than 2. Delete from the initial sub-cluster cluster center set Seed, the initial sub-cluster cluster center set After deletion, there are K sub-cluster centers remaining, K≤K0=n / 10, and the sub-cluster centers also become Repeat this step until the number of subclusters and the position of subclusters in the Seed set converge to obtain the final Step S2.3: Set the sub-cluster centers after convergence Each prototype in As the sub-cluster center, according to the principle of the closest distance, each sample point Belong to the sub-cluster center closest to it , the sample points As the sub-cluster core The cluster members are divided into K subclusters, and the K subclusters are recorded as C1, C2, ..., C K .

3. A data anomaly detection method according to claim 2, characterized in that: The opponent penalty competitive learning formula in step S2.2 is determined as follows: Among them, x i represents the current sample point, Represents the current sample point x i The closest sub-cluster center, Indicates that the current sample point x is found by the first formula above i The nearest sub-cluster center index, The sub-cluster center representing the current reward or punishment Indicates the number of current sub-cluster centers, 0 represents the number of initial sub-cluster centers, represents the penalty coefficient, Expressing defeat to the opponent The intensity of punishment, It means that as the number of sub-cluster centers decreases, the rewards and punishments for each cluster center should be reduced. t represents the number of rewards and punishments. Represents the sub-cluster center The updated value after receiving the reward or penalty from the sample point in the dataset X to be tested for the tth time, Indicates the The next update after being rewarded or penalized.

4. A data anomaly detection method according to claim 3, characterized in that: Learning rate Penalty coefficient The value range is .

5. A data anomaly detection method according to claim 4, characterized in that: Learning rate Penalty coefficient The value of .

6. A data anomaly detection method according to claim 5, characterized in that: The specific process of obtaining the connection strength matrix W between any two subclusters in step S5 is as follows: The improved Floyd algorithm updates and improves a weighted undirected graph by continuously adding subsequent nodes, so that the weight between any two nodes in the weighted undirected graph is maximized. The specific steps include: Step S5.1: The K×K adjacency matrix W0 is a symmetric matrix, let D0=W0, d0( ) represents the first Nodes and The weight between nodes is the size of the weight, the node is the sub-cluster, and the weight is the size of the The subcluster represented by the node and the The connection strength between the subclusters represented by the nodes; Step S5.2: Let k = 1, where k represents taking the kth node in the D0 matrix as a subsequent node and using the subsequent node to expand the path, wherein the value of k ranges from 1 to K; Step S5.3: D k-1 Represents the matrix after the last update, if D k-1 Each element d in the matrix k-1 ( ) satisfies d k-1 ( ) d k-1 ( )>d k-1 ( ) when the The first Nodes and The weight between nodes is d k ( )=d k-1 ( ) d k-1 ( );If D k-1 Each element d in the matrix k-1 ( ) does not satisfy d k-1 ( ) d k-1 ( )>d k-1 ( ), then the The first Nodes and The weight between nodes is d k ( )=d k-1 ( ); until all the For each element in the matrix, get the matrix ; Step S5.4: If k < K, then let k = k + 1, and repeat Step S5.3 until k = K, at which point the matrix is the connection strength matrix between any two sub-clusters, that is = W.

7. A data anomaly detection method according to claim 6, characterized in that: The specific process of step S6 is as follows: Step S6.1: Calculate each sample point in the data set to be tested and the converged sub-cluster center set Each sub-cluster center in The similarity calculation formula of similarity is: in, express The standard deviation of the sub-cluster represented by xi can be obtained according to the calculated similarity to obtain an n×K similarity matrix S, where n represents the number of sample points in the data set to be tested, K represents the K sub-clusters into which the data set to be tested is divided, and S(i, j) = Similarity(xi, mj) represents the similarity between the i-th sample point and the j-th sub-cluster center. Step S6.2: Based on each sample point The relationship between the sample points and the K subclusters is defined as an n×K matrix C, where n represents the number of sample points in the dataset to be tested, and K represents the K subclusters into which the dataset to be tested is divided. belong The subcluster represented by , then C ( ) = 1, if the sample point Not The subcluster represented by , then C ( ) = 0, where j ranges from 1 to K; Step S6.3: Based on each sample point The similarity between each sample point and the center of other sub-clusters is adjusted by the connection strength between the sub-cluster to which it belongs and other sub-clusters, and each sample point in the data set to be tested is obtained. The adjusted n×K similarity matrix to each sub-cluster center prototype , sample points The connection strength between the sub-cluster to which it belongs and other sub-clusters is obtained according to the connection strength matrix W between any two sub-clusters. The adjusted n×K similarity matrix Determined by the following formula: in, Indicates the similarity between the i-th sample point in the data set to be detected and the j-th sub-cluster center, Represents the connection strength weight between the kth sub-cluster and the jth sub-cluster, k is the sample point x i The subcluster to which it belongs can be obtained through the matrix C, that is, k satisfies C(i, k) = 1; Step S6.4: The similarity between each sample point in the dataset to be detected and the center of each sub-cluster can be obtained based on the adjusted n×K similarity matrix Sim, and the anomaly score is calculated based on the similarity. The specific process is as follows: Add each row in the adjusted n×K similarity matrix Sim to obtain an n×1 row vector; then normalize the row vector, subtract the normalized row vector from 1, and obtain the anomaly score corresponding to each sample point in the sample to be detected. Rank the anomaly scores in descending order, and the sample points corresponding to the top anomaly scores are the abnormal data.

8. A data anomaly detection system, characterized in that: The system is based on the method according to any one of claims 1 to 7, comprising: Acquisition module: used to obtain the data set to be tested; Partitioning module: used to divide the data set to be detected into K subclusters using an adaptive partitioning method, and retain the cluster center position of each subcluster, each of the K subclusters contains multiple sample points; The module for calculating the connection strength between adjacent subclusters is used to input all adjacent subclusters and the cluster center positions of the corresponding subclusters in the K subclusters into the algorithm based on the Gaussian mixture model, and output the connection strength between all adjacent subclusters; Obtain weighted undirected graph module: used to construct a weighted undirected graph between adjacent subclusters based on the adjacent relationship between subclusters and the connection strength between adjacent subclusters; A module for obtaining a connection strength matrix between arbitrary subclusters is used to obtain a K*K adjacency matrix W0 based on the weighted undirected graph between the adjacent subclusters, and input the K*K adjacency matrix W0 into the improved Floyd algorithm to obtain a connection strength matrix W between arbitrary subclusters. The connection strength matrix W between arbitrary subclusters is represented as the adjacency matrix of the weighted undirected graph between arbitrary subclusters in the K subclusters. Detection module: Use the similarity strategy to calculate the similarity between each sample point and the K sub-cluster centers to obtain the initial similarity. Adjust the connection strength matrix W between any two sub-clusters by the initial similarity to obtain the adjusted similarity. Calculate the anomaly score based on the adjusted similarity and rank the anomaly scores in descending order. The sample points corresponding to the top anomaly scores are the abnormal data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer stored program is executed by a processor, the data anomaly detection method according to any one of claims 1 to 7 is implemented.

Images