Method and device for realizing abnormal data detection and electronic equipment

Abstract

The invention provides a method and device for achieving abnormal data detection and electronic device.The method comprises the steps that an original data set is generated, and the original data set comprises multiple pieces of first original data with abnormal tags and multiple pieces of second original data without tags; clustering the original data in the original data set to generate a plurality of clusters; under the condition that the missing report proportion of the clustering cluster does not exceed a preset threshold value, marking a label corresponding to the clustering cluster for second original data in the clustering cluster; multiple pieces of original data selected from the original data set form a training set; and training according to the training set to obtain a classification model. Through the method and the device for realizing abnormal data detection and the electronic equipment provided by the embodiment of the invention, a large amount of original data with labels can be obtained, and then the classification model with relatively good detection performance can be trained by utilizing the original data, so that the calculation amount is small, and the real-time performance is high.

Description

Technical Field

[0001] The present invention relates to the technical field of data detection, and in particular, to a method, device, electronic device, and computer-readable storage medium for realizing abnormal data detection. Background Art

[0002] In actual network security anomaly detection work, whether the detection object is traffic data or log data, in most cases, it is unlabeled, and only in a small number of cases is it labeled. For example: relying on the already detected and identified security threats, or security events reported by users, and abnormal events known through post-event notifications, etc., corresponding abnormal labels can be attached to the corresponding traffic or log data. However, these labeled abnormal data only account for a very small number in all the massive data. Most of the traffic or log data is unlabeled, and this unlabeled data includes both normal data and unrecognized abnormal data. Therefore, how to use the existing small amount of labeled data to construct a network anomaly detection model to realize the anomaly detection of unlabeled network traffic or log data and achieve high accuracy and recall rate is a very crucial issue.

[0003] Currently, for the anomaly detection problem with only a small number of labels, most use semi-supervised learning methods, such as generative model algorithms and self-training algorithms. However, the main disadvantages of the current algorithms are, firstly, in the case of a small amount of labeled data, only using the labeled data to train the model is likely to cause overfitting of the model, making the trained detection model have very good detection performance for the existing labeled data, but poor detection performance for new unlabeled data; secondly, for a dataset with a small number of labels, if an unsupervised clustering algorithm is used as the core detection method, then each time a new piece of data is classified, the clustering model needs to be run again, which will bring a large amount of computation for massive traffic or log data. If a batch of new data is accumulated and then the clustering algorithm is run, although the amount of computation can be reduced, there will be a delay, which is not applicable to scenarios with high requirements for detection real-time performance. Summary of the Invention

[0004] To solve the technical problem that the classification model has a poor effect due to the small amount of labeled data, the embodiments of the present invention provide a method, device, electronic device, and computer-readable storage medium for realizing abnormal data detection.

[0005] In a first aspect, the embodiments of the present invention provide a method for realizing abnormal data detection, including:

[0006] Generating an original data set, where the original data set includes a plurality of first original data with abnormal labels and a plurality of second original data without labels;

[0007] Perform clustering processing on the original data in the original dataset to generate multiple clustering clusters, where each clustering cluster includes multiple pieces of the original data;

[0008] Determine the false negative ratio of the clustering cluster, where the false negative ratio is the ratio of the first original data being misclustered;

[0009] When the false negative ratio does not exceed a preset threshold, label the second original data in the clustering cluster with the label corresponding to the clustering cluster;

[0010] Take multiple normal original data and multiple abnormal original data selected from the original dataset as training data, and form a training set containing multiple pieces of the training data;

[0011] Train a classification model for identifying whether data is abnormal based on the training set.

[0012] In a possible implementation, the performing clustering processing on the original data in the original dataset to generate multiple clustering clusters includes:

[0013] Perform clustering processing on the original data in the original dataset, take multiple original data that can be clustered into one category as a normal clustering cluster, and take the other original data except the normal clustering cluster as another abnormal clustering cluster.

[0014] In a possible implementation, the determining the false negative ratio of the clustering cluster includes: determining the number of the first original data in the normal clustering cluster;

[0015] Determine the false negative ratio of the clustering cluster according to the number of the first original data in the normal clustering cluster; there is a positive correlation between the false negative ratio and the number of the first original data in the normal clustering cluster.

[0016] In a possible implementation, the number of the first original data is less than the number of the second original data.

[0017] In a possible implementation, after training a classification model for identifying whether data is abnormal based on the training set, the method further includes:

[0018] Determine a test set containing multiple test data, where the test data is data collected during the current time period;

[0019] Perform the same feature extraction processing on each of the training data and the test data, and extract the multi-dimensional features of the training data and the multi-dimensional features of the test data respectively;

[0020] Determine the first distribution situation and the second distribution situation; the first distribution situation is used to represent the distribution situation of the multi-dimensional features of all the training data in the training set, and the second distribution situation is used to represent the distribution situation of the multi-dimensional features of all the test data in the test set;

[0021] Judge whether the first distribution situation is consistent with the second distribution situation, and update the classification model when the first distribution situation is inconsistent with the second distribution situation.

[0022] In a possible implementation manner, the determining the first distribution situation and the second distribution situation includes:

[0023] Perform clustering processing on the training data in the training set to generate a plurality of first clusters, and each first cluster includes a plurality of the training data;

[0024] Determine the number of different values of each dimension feature of all the training data in the first cluster;

[0025] Generate a first feature dimension matrix for representing the first distribution situation; the element in the i-th row and k-th column of the first feature dimension matrix represents: in the i-th first cluster, the number of different values of the k-th dimension feature;

[0026] Perform clustering processing on the test data in the test set to generate a plurality of second clusters, and each second cluster includes a plurality of the test data;

[0027] Determine the number of different values of each dimension feature of all the test data in the second cluster;

[0028] Generate a second feature dimension matrix for representing the second distribution situation; the element in the j-th row and k-th column of the second feature dimension matrix represents: in the j-th second cluster, the number of different values of the k-th dimension feature.

[0029] In a possible implementation manner, the judging whether the first distribution situation is consistent with the second distribution situation includes:

[0030] Calculate the chi-square value according to the first feature dimension matrix and the second feature dimension matrix, and the chi-square value χ 2 Satisfies:

[0031]

[0032] Wherein, g ik Represents the element in the i-th row and k-th column of the first feature dimension matrix, f jk Represents the element in the j-th row and k-th column of the second feature dimension matrix, m represents the number of the first clusters, n represents the number of the second clusters, and s represents the number of dimensions of the features;

[0033] When the chi-square value χ 2 is greater than the selected critical value, it is determined that the first distribution is inconsistent with the second distribution.

[0034] In a second aspect, an embodiment of the present invention further provides an apparatus for implementing abnormal data detection, including:

[0035] A generation module, configured to generate an original data set, where the original data set includes a plurality of first original data with abnormal labels and a plurality of second original data without labels;

[0036] A clustering module, configured to perform clustering processing on the original data in the original data set to generate a plurality of clustering clusters, where each clustering cluster includes a plurality of the original data;

[0037] A determination module, configured to determine the false negative ratio of the clustering cluster, where the false negative ratio is the ratio of the first original data being mis-clustered;

[0038] A marking module, configured to, when the false negative ratio does not exceed a preset threshold, mark the second original data in the clustering cluster with the label corresponding to the clustering cluster;

[0039] A training module, configured to use a plurality of normal original data and a plurality of abnormal original data selected from the original data set as training data to form a training set including a plurality of the training data; and train a classification model for identifying whether data is abnormal according to the training set.

[0040] In a third aspect, an embodiment of the present invention provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored on the memory and executable on the processor. The transceiver, the memory, and the processor are connected through the bus. When the computer program is executed by the processor, the steps in the method for implementing abnormal data detection according to any one of the above are implemented.

[0041] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method for implementing abnormal data detection according to any one of the above are implemented.

[0042] The method, device, electronic device, and computer-readable storage medium for implementing abnormal data detection provided by the embodiments of the present invention mix the first original data with abnormal labels and the second original data without labels, and perform clustering. Based on the false negative ratio of the clustering clusters, it is determined whether the current clustering result is correct. Furthermore, when the clustering result is correct, corresponding labels can be assigned to the second original data without labels, so as to obtain a large amount of original data with labels. Then, a classification model capable of identifying whether the data is abnormal can be trained using these original data. This method can accurately and quickly label the second original data. After that, using a large amount of original data with labels, a classification model with good detection performance can be obtained, which can accurately identify new unlabeled data. Moreover, compared with unsupervised algorithms, during the identification process, this method does not need to cluster new unlabeled data, has a small amount of computation, and high real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background art, the drawings required for use in the embodiments of the present invention or the background art will be described below.

[0044] Figure 1 The flowchart of a method for implementing abnormal data detection provided by the embodiments of the present invention is shown;

[0045] Figure 2 The flowchart for determining whether to update the classification model in the method for implementing abnormal data detection provided by the embodiments of the present invention is shown;

[0046] Figure 3 The structural schematic diagram of a device for implementing abnormal data detection provided by the embodiments of the present invention is shown;

[0047] Figure 4 The structural schematic diagram of an electronic device for executing the method for implementing abnormal data detection provided by the embodiments of the present invention is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] The embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.

[0049] Figure 1 The flowchart of a method for implementing abnormal data detection provided by the embodiments of the present invention is shown. As Figure 1 shown, the method includes:

[0050] Step 101: Generate an original data set, which includes a plurality of first original data with abnormal labels and a plurality of second original data without labels.

[0051] In the embodiments of the present invention, an initial data set, i.e., a raw data set, is generated using data that originally has anomaly tags and data that originally has no tags, and the data in the raw data set are all referred to as raw data; for convenience of distinction, the raw data with anomaly tags is referred to as first raw data, and the raw data without tags is referred to as second raw data. For example, the data may specifically be traffic data, log data, etc. in a network. By using an existing anomaly library that has been labeled (for example, relying on security threats that have been detected and identified, or security events reported by users, as well as anomaly events known from after-the-fact notifications, etc., corresponding traffic or log data can be labeled with anomaly tags to form an anomaly library), multiple first raw data can be obtained, and multiple second raw data without tags can be obtained by means such as collecting from the network.

[0052] Generally, the number of first raw data is small, while the number of second raw data is large, that is, the number of first raw data is less than the number of second raw data. For example, the second raw data has a larger data magnitude, and its number exceeds ten times the number of first raw data. In the embodiments of the present invention, using a small amount of first raw data with anomaly tags can achieve labeling of a large amount of second raw data without tags. For example, labeling the second raw data as normal data (normal tag) or as anomaly data (anomaly tag). Generally, most of the second raw data is normal data, and only a small part of the second raw data is anomaly data.

[0053] Step 102: Perform clustering processing on the raw data in the raw data set to generate multiple clustering clusters, and each clustering cluster includes multiple raw data.

[0054] After generating a raw data set mixed with first raw data and second raw data, clustering is performed on the raw data therein (both first raw data and second raw data are raw data), so that multiple raw data can be divided into multiple clusters, that is, clustering clusters; each clustering cluster also contains multiple raw data. Among them, a clustering cluster may contain multiple first raw data, may contain multiple second raw data, or may contain at least one first raw data and at least one second raw data, depending on the actual situation. Among them, the clustering algorithm used for clustering processing is not limited to a specific method, and common algorithms such as density clustering and kmeans can be used.

[0055] Since normal raw data and anomaly raw data have different characteristics, when clustering the raw data, normal raw data and anomaly raw data can be distinguished.

[0056] For example, the original data can be clustered into multiple categories, and one of the categories is the discrete points in the original data; since most traffic data is normal, generally only the abnormal traffic data is the discrete point, so the clustering cluster composed of discrete points represents the abnormal original data, and other clustering clusters represent the normal original data. Or, the original data can also be clustered into two categories, one of which is the clustering cluster containing normal original data (which can be simply referred to as the normal clustering cluster), and the other is the clustering cluster containing abnormal original data (which can be simply referred to as the abnormal clustering cluster). For example, after clustering the original data into two categories, the cluster with a small number of original data and most of the first original data is used as the clustering cluster containing the original data.

[0057] Step 103: Determine the false negative rate of the clustering cluster, where the false negative rate is the proportion of the first original data that is misclustered.

[0058] Step 104: When the false negative rate does not exceed the preset threshold, label the second original data in the clustering cluster with the label corresponding to the clustering cluster.

[0059] In the embodiments of the present invention, when clustering the original data into multiple clustering clusters, generally, whether the original data has labels or not, the normal original data and the abnormal original data will not be clustered into one category; for example, the normal original data will be clustered into one category, and the abnormal original data will be clustered into another category. Since the original data contains the first original data with abnormal labels itself, by judging whether the clustering results of multiple first original data are correct, it can be determined whether the clustering results of the entire original data set are correct, that is, whether the multiple clustering clusters obtained by clustering are appropriate. In the embodiments of the present invention, the false negative rate is used to judge whether the clustering results are correct.

[0060] Specifically, the false negative rate represents the proportion of the first original data that is misclustered, that is, the proportion of the first original data that originally belongs to the abnormal data but is clustered as normal data; the larger the false negative rate, the more inaccurate the current clustering results. In the embodiments of the present invention, a threshold is set for the false negative rate, that is, the preset threshold. If the false negative rate exceeds the preset threshold, it means that the current clustering results are inaccurate and need to be reclustered, such as changing the parameters used in the clustering algorithm, etc.; if the false negative rate does not exceed the preset threshold, it can be shown that the current clustering results are relatively accurate, so that the second original data without labels in the clustering cluster can be labeled with the label corresponding to the clustering cluster, so that the second original data also has a label, that is, the second original data can also be used for supervised training. For example, set the normal label for the second original data in the normal clustering cluster and set the abnormal label for the second original data in the abnormal clustering cluster.

[0061] Step 105: Use multiple normal original data and multiple abnormal original data selected from the original dataset as training data to form a training set containing multiple training data.

[0062] Step 106: Train a classification model for identifying whether data is abnormal based on the training set.

[0063] In the embodiment of the present invention, through the above steps 101-104, appropriate labels can be marked on the second original data that was originally unlabeled, so that a large amount of labeled original data can be obtained, that is, the original data in the original dataset are all labeled. In this case, a sufficient number of original data can be selected as training data to form a training set, and thus the classification model can be trained using this training set, so that the trained classification model can identify whether the data is abnormal. Moreover, the training set needs to include original data with normal labels and also original data with abnormal labels; for example, all the original data can be used as training data to form this training set; or, the training set can also be generated only based on multiple second original data, and the present embodiment does not limit this. Among them, any supervised learning algorithm can be used for training, such as using a support vector machine, a k-nearest neighbor algorithm, or a decision tree algorithm for training, or neural network-related algorithms can also be used for training.

[0064] After training this classification model, the classification model can be used to classify the currently collected data to detect whether the data is abnormal, so as to achieve abnormal data detection. For example, the above original data is log data, and a new log data A is currently collected. If it is necessary to determine whether the log data A is abnormal, the log data A can be input into the classification model, and based on the result of the classification model, it can be determined whether the log data A is abnormal.

[0065] A method for implementing abnormal data detection provided by the embodiment of the present invention mixes the first original data with abnormal labels and the second original data without labels, and performs clustering based on an unsupervised clustering method. Based on the false negative rate of the clustering clusters, it is determined whether the current clustering result is correct. Furthermore, when the clustering result is correct, corresponding labels can be marked on the second original data without labels, so as to obtain a large amount of original data with labels. Then, these original data can be used to train a classification model that can identify whether the data is abnormal. This method can label the second original data relatively accurately and quickly. After that, using a large amount of labeled original data, a classification model with better detection performance can be obtained, which can relatively accurately identify new unlabeled data; and, compared with unsupervised algorithms, in the recognition process, this method does not need to cluster new unlabeled data either, with a small amount of computation and high real-time performance.

[0066] Optionally, step 102 above, "cluster the original data in the original dataset to generate multiple clustering clusters", may specifically include: clustering the original data in the original dataset, taking multiple original data that can be clustered into one category as a normal clustering cluster, and taking the other original data except the normal clustering cluster as another abnormal clustering cluster.

[0067] In the embodiment of the present invention, when clustering the original data, it is divided into two categories. One category is the normal clustering cluster, that is, the normal clustering cluster; the other category is the outliers in the original data, and the outliers are taken as another clustering cluster. In the field of traffic, since most traffic data is normal, the outliers in the clustering result are considered abnormal, and the clustering cluster formed by the outliers is the abnormal clustering cluster, that is, the abnormal clustering cluster; while the clusters that can be normally clustered are considered normal, that is, the normal clustering cluster. Since it is uncertain whether the abnormal outliers found by this clustering method are reliable, the embodiment of the present invention relies on those known and determined abnormal first original data to help adjust the clustering parameters to make the clustering result more accurate.

[0068] Optionally, when it is necessary to determine the false negative rate, step 103 above, "determine the false negative rate of the clustering cluster" may include the following steps A1 - A2:

[0069] Step A1: Determine the number of the first original data in the normal clustering cluster.

[0070] Step A2: Determine the false negative rate of the clustering cluster according to the number of the first original data in the normal clustering cluster; there is a positive correlation between the false negative rate and the number of the first original data in the normal clustering cluster.

[0071] In the embodiment of the present invention, in the case of correct clustering, all the first original data with abnormal labels should belong to the abnormal clustering cluster, that is, the number of the first original data in the normal clustering cluster is 0; but in practice, it is easy to have inaccurate clustering, that is, classifying the abnormal first original data into the normal clustering cluster, and the number of the first original data in the normal clustering cluster is greater than 0, resulting in false negatives. The number of the first original data in the normal clustering cluster can represent the severity of the false negatives; the larger this number is, the more serious the false negatives are.

[0072] Specifically, in this embodiment, the false negative rate is used to represent the severity of the false negatives, and there is a positive correlation between the false negative rate and the number of the first original data in the normal clustering cluster; for example, false negative rate = the number of the first original data in the normal clustering cluster / the total number of the first original data, or false negative rate = the number of the first original data in the normal clustering cluster / the total number of the original data in the original dataset, etc.

[0073] Optionally, to avoid clustering a large amount of actually normal second original data into an abnormal clustering cluster, the number of first original data in this normal clustering cluster can be greater than 0, that is, there is first original data in this normal clustering cluster, but the number of this first original data cannot be too large.

[0074] Since the network state changes in real time and the attack means of the attacker will also be upgraded, resulting in new types of attacks, these factors may all cause concept drift in the data input into the classification model; for an already constructed classification model, as the input data undergoes concept drift, its detection performance will decline. To enable the classification model to adapt to new types of attacks, it is very important to adjust and update the classification model in a timely manner. If the classification model is updated frequently, although the performance of the classification model can be guaranteed, there is an easy problem of updating the classification model when new types of attacks do not occur; on the contrary, if the classification model is updated at a relatively long interval, new types of attacks are likely to be missed, and the performance of the classification model is poor for a long time.

[0075] In the embodiment of the present invention, by determining whether the data distribution changes to determine whether a new type of attack appears, it is possible to more reasonably determine whether the classification model needs to be updated. Specifically, refer to Figure 2 As shown, after step 106 "training a classification model for identifying whether data is abnormal according to a training set", the method further includes:

[0076] Step 107: Determine a test set including a plurality of test data, and the test data is data collected during the current time period.

[0077] In the embodiment of the present invention, the data collected during the current time period is used as test data, thereby forming a test set including a plurality of test data. For example, a sliding time window can be set, the end of the sliding time window is the current moment, the time period corresponding to the sliding time window is the current time period, and the data collected within the sliding time window can all be used as test data. Among them, the test data and the training data used to train the classification model are of the same type. For example, both are log data or both are traffic data.

[0078] Among them, both the training set and the test set include a plurality of data. Represent the training set as Train and the test set as Test, then the training set Train = {d i , i = 1, 2,..., D}, the test set Test = {t j , j = 1, 2,..., T}; where d i represents the i-th training data, t j represents the j-th test data, D is the number of training data, and T is the number of test data.

[0079] Step 108: Perform the same feature extraction process on each training data and test data, and extract the multi-dimensional features of the training data and the multi-dimensional features of the test data respectively.

[0080] In the embodiments of the present invention, each training data and test data are processed in the same feature extraction manner, so that the features of each training data and each test data can be extracted; the features of the training data and the test data are both multi-dimensional, and since the same feature extraction process is adopted, the feature dimensions of the two are also the same. For example, for each training data d i , s-dimensional features can be extracted; for each test data t j , s-dimensional features can also be extracted; s is the dimension of the features, and s>1.

[0081] Step 109: Determine the first distribution and the second distribution; the first distribution is used to represent the distribution of the multi-dimensional features of all training data in the training set, and the second distribution is used to represent the distribution of the multi-dimensional features of all test data in the test set.

[0082] Step 110: Determine whether the first distribution is consistent with the second distribution. If the first distribution is inconsistent with the second distribution, update the classification model.

[0083] In the embodiments of the present invention, after determining the s-dimensional features of each training data d i , the distribution of these features, that is, the first distribution, can be further determined. The first distribution can represent the distribution of the training data d i , that is, d i ~Φ1, where Φ1 represents the first distribution. Similarly, after determining the s-dimensional features of each test data t j , the distribution of these features, that is, the second distribution, can also be further determined. The second distribution can represent the distribution of the test data t j , that is, t j ~Φ2, where Φ2 represents the second distribution. If there is no concept drift, the distribution difference between the first distribution Φ1 and the second distribution Φ2 of the two kinds of data is relatively small, and the two are basically consistent; if there is concept drift, the distribution difference between the first distribution Φ1 and the second distribution Φ2 of the two kinds of data is relatively large.

[0084] Specifically, in the embodiments of the present invention, it is determined whether the first distribution situation is consistent with the second distribution situation to determine whether the difference between the first distribution situation Φ1 and the second distribution situation Φ2 is small enough; if the first distribution situation is inconsistent with the second distribution situation, it can be considered that the difference between the two is large, that is, compared with the training data used to train the classification model before, the current test data is quite different. At this time, there is probably a new type of attack situation, so the classification model can be updated at this time. For example, these test data can be used as the second raw data without labels, and the above steps 101-106 can be re-executed to obtain an updated classification model, which can accurately detect the current data.

[0085] Optionally, the above step 109 "determine the first distribution situation and the second distribution situation" may specifically include the following steps B1-B6:

[0086] Step B1: Perform clustering processing on the training data in the training set to generate multiple first clusters, and each first cluster includes multiple training data.

[0087] Step B2: Determine the number of different values of each dimension feature of all training data in the first cluster.

[0088] Step B3: Generate a first feature dimension matrix for representing the first distribution situation; the element in the i-th row and k-th column of the first feature dimension matrix represents: in the i-th first cluster, the number of different values of the k-th dimension feature.

[0089] In the embodiments of the present invention, the multi-dimensional features extracted from the training data are used as the attributes for clustering, and the training data in the training set is clustered, so that the training data can be divided into multiple clusters. In the embodiments of the present invention, these clusters are all called first clusters; correspondingly, each first cluster contains multiple training data. For example, cluster all the training data in the training set Train to form m first clusters C i , i = 1, 2, 3,..., m; where, for any two different first clusters, the data within the clusters does not intersect, that is, the training set Train can also be expressed as: Train = C1 ∪ C2 ∪... C i ... ∪ C m (i = 1, 2, 3,..., m).

[0090] For each first cluster, taking the dimension of the feature as a unit, determine the number of different values corresponding to each dimension of the feature respectively. The number of different values refers to: for a certain dimension of the feature, the number of different values of this dimension of the feature in all training data within a first cluster. Since the training data within the same first cluster generally have relatively similar features, the number of different values is generally less than the number of training data within this first cluster. For example, a first cluster contains 10 training data, and the k-th features of these 10 training data are respectively: a, a, b, b, c, d, d, d, e, e. That is, for the k-th feature, these 10 training data have a total of five features, namely a, b, c, d, e, that is, there are 5 different values, and the number of different values of the k-th feature is 5.

[0091] Based on this, for each first cluster, the number of different values corresponding to each dimension of the feature can be determined. In the embodiments of the present invention, all the numbers of different values corresponding to the training set are represented by a feature dimension matrix; for the convenience of description, the feature dimension matrix corresponding to the training set is called the first feature dimension matrix. Each row of this first feature dimension matrix corresponds to a first cluster, and each column corresponds to one dimension of the feature. The element in the i-th row and the k-th column of this first feature dimension matrix represents: in the i-th first cluster, the number of different values of the k-th dimension of the feature.

[0092] For example, each training data has s-dimensional features, and the number of first clusters is m, then this first feature dimension matrix is an m×s matrix, and the element g in the i-th row and the k-th column of this first feature dimension matrix ik , that is, it represents the number of different values of the k-th dimension of the feature in the i-th first cluster; where i = 1, 2, 3, …, m, k = 1, 2, 3, …, s.

[0093] Step B4: Perform clustering processing on the test data in the test set to generate multiple second clusters, and each second cluster includes multiple test data.

[0094] Step B5: Determine the number of different values of each dimension of the feature for all the test data in the second cluster.

[0095] Step B6: Generate a second feature dimension matrix for representing the second distribution situation; the element in the j-th row and the k-th column of the second feature dimension matrix represents: in the j-th second cluster, the number of different values of the k-th dimension of the feature.

[0096] In the embodiments of the present invention, similar to the above steps B1 - B3, the multi-dimensional features extracted from the test data are used as the attributes for clustering, and the test data in the test set are clustered, so that the test data can be divided into multiple clusters. In the embodiments of the present invention, these clusters are all called second clusters; correspondingly, each second cluster contains multiple test data. For example, cluster all the test data in the test set Test to form n second clusters c j, j = 1, 2, 3, …, n; where, for any two different second clusters, the data within the clusters do not intersect, that is, the test set Test can also be expressed as: Test = c1 ∪ c2 ∪... c j ... ∪ c n (j = 1, 2, 3, …, n).

[0097] For each second cluster, taking the dimension of the feature as the unit, determine the number of different values corresponding to each dimension of the feature respectively. Similarly, the number of different values refers to: for a certain dimension of the feature, the number of different values of this dimension of the feature of all test data within a second cluster. Since the test data within the same second cluster generally have relatively similar features, the number of different values is generally less than the number of test data within this second cluster.

[0098] Based on this, the number of different values corresponding to each dimension of the feature within each second cluster can be determined. In the embodiment of the present invention, the feature dimension matrix represents all the different values corresponding to the test set; for convenience of description, the feature dimension matrix corresponding to the test set is called the second feature dimension matrix. Each row of this second feature dimension matrix corresponds to a second cluster, and each column corresponds to one dimension of the feature. The element in the j-th row and k-th column of this second feature dimension matrix represents: the number of different values of the k-th dimension of the feature in the j-th second cluster.

[0099] For example, each test data has s-dimensional features, and the number of second clusters is n, then this second feature dimension matrix is an n×s matrix. The element f jk in the j-th row and k-th column of this second feature dimension matrix represents the number of different values of the k-th dimension of the feature in the j-th second cluster; where, j = 1, 2, 3, …, n, k = 1, 2, 3, …, s.

[0100] In the embodiment of the present invention, the number of clusters of the first cluster of the training data obtained by clustering and the second cluster of the test data may be different, that is, the m value and the n value may be different. However, because the training data and the test data are of the same type, their formats are the same, and the same feature extraction process is used, so the feature dimensions are the same, both being s-dimensional.

[0101] Optionally, the above step 110 "determine whether the first distribution situation is consistent with the second distribution situation" may specifically include the following steps C1 - C2:

[0102] Step C1: Calculate the chi-square value according to the first feature dimension matrix and the second feature dimension matrix. The chi-square value χ 2 satisfies:

[0103]

[0104] where, g ikrepresents the element in the \(i\)-th row and \(k\)-th column of the first feature dimension matrix, \(f\) jk represents the element in the \(j\)-th row and \(k\)-th column of the second feature dimension matrix, \(m\) represents the number of the first clusters, \(n\) represents the number of the second clusters, and \(s\) represents the dimension of the features.

[0105] Step C2: When the chi-square value \(\chi\) 2 is greater than the selected critical value, it is determined that the first distribution is inconsistent with the second distribution.

[0106] In the embodiments of the present invention, the independence chi-square distribution test is used to determine whether two feature dimension matrices are consistent. Specifically, the chi-square value \(\chi\) 2 is calculated based on the above formula (1), where \(g\) ik , \(f\) jk are the elements in the first feature dimension matrix and the second feature dimension matrix respectively, that is, the number of different values; the above formula (1) is equivalent to a three-layer loop operation. The innermost loop is used to compare the similarity of the feature elements corresponding to different clusters in two different feature dimension matrices, and the outer two loops are used to compare the similarity between a certain cluster in one feature dimension matrix and each cluster in the other feature dimension clustering (that is, each cluster in one matrix needs to be compared with all clusters in the other matrix).

[0107] Wherein, in the above steps B1 and B4, when performing the clustering process, the distance from each sample point (training data or test data) in the cluster to the centroid can be controlled to ensure that the number of sample points in each cluster is appropriate, so as to ensure the uniform density of the partition and at the same time meet the minimum sample number for the chi-square test.

[0108] After determining the chi-square value \(\chi\) 2 , the required critical value can be selected based on the chi-square test method. Specifically, the process of determining the critical value may include: querying the chi-square distribution table according to the degrees of freedom and the preset quantile to determine the critical value; the degrees of freedom \(df\) satisfies: \(df=(n - 1)(s - 1)\). The quantile can be 0.05 for example.

[0109] Specifically, the corresponding critical value in the chi-square distribution table can be obtained by querying according to the degrees of freedom \(df\) and the quantile, and the calculated chi-square value \(\chi\) 2 is compared with the critical value obtained by querying in the chi-square distribution table. If the chi-square value \(\chi\) 2 is greater than the critical value, it is considered that concept drift has occurred in the data within the current time window and model update is required.

[0110] The method for detecting abnormal data provided by an embodiment of the present invention can quickly and easily determine whether the current test data contains a new type of attack by determining whether the first distribution is consistent with the second distribution, thereby determining whether the classification model needs to be updated. By forming a feature dimension matrix using different values for different clusters and different dimensional features, while being able to compare whether the two feature dimension matrices are consistent, it can also more comprehensively retain the attributes that can be represented by each dimensional feature, and can more accurately restore and determine the consistency of the distribution between the training set and the test set.

[0111] The above describes in detail the method for implementing abnormal data detection provided by an embodiment of the present invention. This method can also be implemented by a corresponding device. The following describes in detail the device for implementing abnormal data detection provided by an embodiment of the present invention.

[0112] Figure 3 FIG. 1 shows a schematic diagram of a structure of a device for detecting abnormal data provided by an embodiment of the present invention. Figure 3 As shown, the device for realizing abnormal data detection includes:

[0113] A generating module 31 is configured to generate an original data set, wherein the original data set includes a plurality of first original data with abnormal labels and a plurality of second original data without labels;

[0114] A clustering module 32 is configured to perform clustering processing on the original data in the original data set to generate a plurality of clusters, each of which includes a plurality of the original data;

[0115] A determination module 33 is configured to determine a false negative ratio of the clusters, where the false negative ratio is a ratio of the first original data that is incorrectly clustered;

[0116] A marking module 34 is configured to mark the second original data in the cluster with a label corresponding to the cluster when the underreporting ratio does not exceed a preset threshold;

[0117] The training module 35 is used to use multiple normal original data and multiple abnormal original data selected from the original data set as training data to form a training set containing multiple training data; and to obtain a classification model for identifying whether the data is abnormal based on the training set.

[0118] In a possible implementation, the clustering module 32 performs clustering processing on the original data in the original data set to generate a plurality of clusters, including:

[0119] Perform clustering processing on the original data in the original dataset, and use multiple pieces of original data that can be clustered into one category as a normal clustering cluster, and use other original data except the normal clustering cluster as another abnormal clustering cluster.

[0120] In a possible implementation manner, the determining module 33 determines the false negative ratio of the clustering cluster, including:

[0121] Determine the quantity of the first original data in the normal clustering cluster;

[0122] Determine the false negative ratio of the clustering cluster according to the quantity of the first original data in the normal clustering cluster; there is a positive correlation between the false negative ratio and the quantity of the first original data in the normal clustering cluster.

[0123] In a possible implementation manner, in the original dataset, the quantity of the first original data is less than the quantity of the second original data.

[0124] In a possible implementation manner, the device further includes a judging module. After the training module 35 trains a classification model for identifying whether data is abnormal according to the training set, the judging module is used for:

[0125] Determine a test set including multiple test data, where the test data is data collected during the current time period;

[0126] Perform the same feature extraction processing on each of the training data and the test data, and respectively extract the multi-dimensional features of the training data and the multi-dimensional features of the test data;

[0127] Determine a first distribution situation and a second distribution situation; the first distribution situation is used to represent the distribution situation of the multi-dimensional features of all the training data in the training set, and the second distribution situation is used to represent the distribution situation of the multi-dimensional features of all the test data in the test set;

[0128] Judge whether the first distribution situation is consistent with the second distribution situation, and update the classification model when the first distribution situation is inconsistent with the second distribution situation.

[0129] In a possible implementation manner, the judging module determines the first distribution situation and the second distribution situation, including:

[0130] Perform clustering processing on the training data in the training set to generate multiple first clusters, and each first cluster includes multiple pieces of the training data;

[0131] Determine the number of different values of each dimension feature of all the training data in the first cluster;

[0132] Generate a first feature dimension matrix for representing the first distribution situation; the element in the i-th row and k-th column of the first feature dimension matrix represents: in the i-th first cluster, the number of different values of the k-th dimension feature;

[0133] Perform clustering processing on the test data in the test set to generate a plurality of second clusters, where the second clusters include a plurality of the test data;

[0134] Determine, for each dimension feature of all the test data in the second cluster, the number of different values;

[0135] Generate a second feature dimension matrix for representing the second distribution situation; the element in the j-th row and k-th column of the second feature dimension matrix represents: in the j-th second cluster, the number of different values of the k-th dimension feature.

[0136] In a possible implementation manner, the determining module determines whether the first distribution situation is consistent with the second distribution situation, including:

[0137] Calculate a chi-square value according to the first feature dimension matrix and the second feature dimension matrix, and the chi-square value χ 2 Satisfies:

[0138]

[0139] where g ik represents the element in the i-th row and k-th column of the first feature dimension matrix, and f jk represents the element in the j-th row and k-th column of the second feature dimension matrix, m represents the number of the first clusters, n represents the number of the second clusters, and s represents the number of dimensions of the features;

[0140] When the chi-square value χ 2 is greater than the selected critical value, determine that the first distribution situation is inconsistent with the second distribution situation.

[0141] In a possible implementation manner, the determining module is further configured to:

[0142] Query the chi-square distribution table according to the degrees of freedom and a preset quantile to determine the critical value; the degrees of freedom df satisfies: df = (n - 1)(s - 1).

[0143] An embodiment of the present invention further provides a computer storage medium, where the computer storage medium stores computer-executable instructions, which include a program for executing the method for global control permissions described above, and the computer-executable instructions can execute the methods in any of the above method embodiments.

[0144] Among them, the computer storage medium can be any available medium or data storage device accessible by a computer, including but not limited to magnetic memories (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical memories (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor memories (such as ROM, EPROM, EEPROM, non-volatile memories (NANDFLASH), solid state drives (SSD)), etc.

[0145] Figure 4 The block diagram of a structure of an electronic device showing another embodiment of the present invention is presented. The electronic device 1100 can be a host server with computing capabilities, a personal computer PC, or a portable computer or terminal, etc. The specific embodiments of the present invention do not limit the specific implementation of the electronic device.

[0146] The electronic device 1100 includes at least one processor 1110, a communication interface 1120, a memory 1130, and a bus 1140. Among them, the processor 1110, the communication interface 1120, and the memory 1130 complete communication with each other through the bus 1140.

[0147] The communication interface 1120 is used to communicate with network elements, where the network elements include, for example, a virtual machine management center, shared storage, etc.

[0148] The processor 1110 is used to execute programs. The processor 1110 may be a central processing unit CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0149] The memory 1130 is used for executable instructions. The memory 1130 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory. The memory 1130 may also be a memory array. The memory 1130 may also be partitioned, and the partitions may be combined into virtual volumes according to certain rules. The instructions stored in the memory 1130 can be executed by the processor 1110, enabling the processor 1110 to execute the method of global control authority in any of the above method embodiments.

[0150] The embodiments of the present invention describe the provided methods, devices, and electronic devices through flowcharts and / or block diagrams.

[0151] It should be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing device to produce a machine, and these computer-readable program instructions, when executed by a computer or other programmable data processing device, produce a device for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0152] These computer-readable program instructions can also be stored in a computer-readable storage medium that enables a computer or other programmable data processing device to work in a specific manner. In this way, the instructions stored in the computer-readable storage medium produce an instruction device product that includes instructions for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0153] The computer-readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device, causing a series of operational steps to be executed on the computer, other programmable data processing device, or other device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing device can provide a process for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0154] As described above, this is only the specific implementation manner of the embodiments of the present invention, but the protection scope of the embodiments of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the embodiments of the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the embodiments of the present invention. Therefore, the protection scope of the embodiments of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for implementing abnormal data detection, characterized in that, Including: Generating an original data set, where the original data set includes a plurality of first original data with anomaly labels and a plurality of second original data without labels; Performing clustering processing on the original data in the original data set to generate a plurality of clustering clusters, where each clustering cluster includes a plurality of the original data; Determining the false negative ratio of the clustering cluster, where the false negative ratio is the ratio of the first original data being clustered incorrectly; When the false negative ratio does not exceed a preset threshold, labeling the second original data in the clustering cluster with the label corresponding to the clustering cluster; Taking a plurality of normal original data and a plurality of abnormal original data selected from the original data set as training data to form a training set including a plurality of the training data; Training a classification model for identifying whether data is abnormal based on the training set.

2. The method according to claim 1, characterized in that, The performing clustering processing on the original data in the original data set to generate a plurality of clustering clusters includes: Performing clustering processing on the original data in the original data set, taking a plurality of original data that can be clustered into one category as a normal clustering cluster, and taking the other original data except the normal clustering cluster as another abnormal clustering cluster.

3. The method according to claim 2, characterized in that, The determining the false negative ratio of the clustering cluster includes: Determining the number of the first original data in the normal clustering cluster; Determining the false negative ratio of the clustering cluster according to the number of the first original data in the normal clustering cluster; there is a positive correlation between the false negative ratio and the number of the first original data in the normal clustering cluster.

4. The method according to claim 1, characterized in that In the original data set, the number of the first original data is less than the number of the second original data.

5. The method according to any one of claims 1 to 4, characterized in that, After training the classification model for identifying whether data is abnormal based on the training set, it further includes: Determining a test set including a plurality of test data, where the test data is data collected in the current time period; Performing the same feature extraction processing on each of the training data and the test data, respectively extracting the multi-dimensional features of the training data and the multi-dimensional features of the test data; Determining a first distribution situation and a second distribution situation; the first distribution situation is used to represent the distribution situation of the multi-dimensional features of all the training data in the training set, and the second distribution situation is used to represent the distribution situation of the multi-dimensional features of all the test data in the test set; Judging whether the first distribution situation is consistent with the second distribution situation, and updating the classification model when the first distribution situation is inconsistent with the second distribution situation.

6. The method according to claim 5, wherein The determining the first distribution situation and the second distribution situation includes: Performing clustering processing on the training data in the training set to generate a plurality of first clusters, where each first cluster includes a plurality of the training data; Determining the number of different values of each dimension feature of all the training data in the first cluster; Generating a first feature dimension matrix for representing the first distribution situation; the element in the i-th row and k-th column of the first feature dimension matrix represents: in the i-th first cluster, the number of different values of the k-th dimension feature; Perform clustering processing on the test data in the test set to generate a plurality of second clusters, where each second cluster includes a plurality of the test data; Determine the number of different values of each dimension feature of all the test data in the second cluster; Generate a second feature dimension matrix for representing the second distribution; the element in the j-th row and k-th column of the second feature dimension matrix represents: in the j-th second cluster, the number of different values of the k-th dimension feature.

7. The method according to claim 6, wherein The determination of whether the first distribution is consistent with the second distribution includes: Calculate the chi-square value according to the first feature dimension matrix and the second feature dimension matrix, where the chi-square value χ 2 satisfies: where g ik represents the element in the i-th row and k-th column of the first feature dimension matrix, f jk represents the element in the j-th row and k-th column of the second feature dimension matrix, m represents the number of the first clusters, n represents the number of the second clusters, and s represents the dimension of the features; When the chi-square value χ 2 is greater than the selected critical value, it is determined that the first distribution is inconsistent with the second distribution.

8. An apparatus for implementing abnormal data detection, characterized in that Includes: A generation module for generating an original data set, where the original data set includes a plurality of first original data with anomaly labels and a plurality of second original data without labels; A clustering module for performing clustering processing on the original data in the original data set to generate a plurality of clustering clusters, where each clustering cluster includes a plurality of the original data; A determination module for determining the false negative rate of the clustering cluster, where the false negative rate is the proportion of the first original data that is mis-clustered; A marking module for, when the false negative rate does not exceed a preset threshold, marking the second original data in the clustering cluster with the label corresponding to the clustering cluster; A training module for using a plurality of normal original data and a plurality of abnormal original data selected from the original data set as training data to form a training set including a plurality of the training data; training a classification model for identifying whether data is abnormal according to the training set.

9. An electronic device, comprising a bus, a transceiver, a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the transceiver, the memory, and the processor are connected via the bus, characterized in that, When the computer program is executed by the processor, it implements the steps in the method for detecting abnormal data as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps in the method for detecting abnormal data as described in any one of claims 1 to 7.

Images