Abnormal data detection model construction method, abnormal data detection method and system

Abstract

The present invention provides a method for constructing an abnormal data detection model, an abnormal data detection method and a system, wherein the method uses unlabeled data and abnormal data carrying abnormal labels to train a neural network model to obtain a final abnormal data detection model. The abnormal data in the massive data is detected by the abnormal detection model to improve the detection accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for constructing an abnormal data detection model, an abnormal data detection method and a system. Background Art

[0002] The task of abnormal data detection aims to identify data objects that have obvious deviations from the majority of data objects. The detection of abnormal data can be used to detect network attacks and abnormal credit card transactions.

[0003] At present, traditional anomaly data detection algorithms (such as LOF, iForest and SVDD, etc.) are usually used to detect anomaly data. However, due to the data dimensionality disaster and high computational complexity, traditional anomaly data detection methods cannot accurately detect anomaly data from high-dimensional data sets and massive data, and the detection accuracy is poor. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method for constructing an abnormal data detection model, an abnormal data detection method and a system to solve the problems of poor detection accuracy in traditional abnormal data detection methods.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a method for constructing an abnormal data detection model, the method comprising:

[0007] Determine a plurality of first sample data and a plurality of second sample data from a dedicated anomaly detection data set, wherein the first sample data is unlabeled data, and the second sample data is abnormal data carrying an abnormality label;

[0008] Clustering the second sample data to obtain at least one abnormal data cluster center;

[0009] Performing a priori estimation on the distribution of the plurality of first sample data to obtain an isolation score corresponding to each of the first sample data;

[0010] Calculating the similarity between the first sample data and the second sample data according to the distance between the first sample data and each of the abnormal data cluster centers to obtain a distance score for the first sample data;

[0011] Calculating a priori anomaly scores corresponding to the first sample data using the isolation scores and the distance scores corresponding to the first sample data;

[0012] Determine a priori anomaly score average and a priori anomaly score standard deviation based on a plurality of prior anomaly scores corresponding to the first sample data;

[0013] The prior anomaly score mean and the prior anomaly score standard deviation are used as partial hyperparameters of a preset neural network model, and the neural network model is trained using the first sample data, the prior anomaly score corresponding to the first sample data, and the second sample data to obtain a final abnormal data detection model.

[0014] Preferably, determining a plurality of first sample data and a plurality of second sample data from the anomaly detection dedicated data set includes:

[0015] Extracting a preset percentage of data from a special data set for anomaly detection to obtain a model training set, wherein the special data set for anomaly detection includes normal data with normal labels and abnormal data with abnormal labels;

[0016] Determining the amount of noise according to the amount of normal data in the model training set;

[0017] Based on the amount of noise, randomly extract abnormal data from the model training set, and randomly divide the randomly extracted abnormal data into multiple groups of abnormal data, each group of abnormal data includes two abnormal data;

[0018] Perform data feature swapping of preset dimensions on each group of abnormal data to obtain noise data;

[0019] Randomly mixing the noise data with normal data in the model training set to obtain a plurality of first sample data;

[0020] A plurality of abnormal data as second sample data are randomly extracted from the model training set.

[0021] Preferably, performing a priori estimation on the distribution of the plurality of first sample data to obtain the isolation score corresponding to each of the first sample data includes:

[0022] An isolation forest algorithm is used to perform a priori estimation on the distribution of the plurality of first sample data to obtain an isolation score corresponding to each of the first sample data.

[0023] Preferably, calculating the similarity between the first sample data and the second sample data according to the distance between the first sample data and each of the abnormal data cluster centers to obtain the distance score of the first sample data includes:

[0024] For each first sample data, determining the abnormal data cluster center closest to the first sample data according to the distance between the first sample data and each abnormal data cluster center;

[0025] For each first sample data, the similarity between the first sample data and the second sample data is calculated using the distance between the first sample data and the abnormal data cluster center closest to the first sample data to obtain a distance score of the first sample data.

[0026] Preferably, the prior anomaly score mean value and the prior anomaly score standard deviation are used as part of the hyperparameters of a preset neural network model, and the neural network model is trained using the first sample data, the prior anomaly score corresponding to the first sample data, and the second sample data to obtain a final abnormal data detection model, including:

[0027] Performing data dimension reduction and feature extraction on the first sample data and the second sample data to obtain a new data representation;

[0028] The prior anomaly score mean and the prior anomaly score standard deviation are used as partial hyperparameters of a preset neural network model, and the neural network model is trained using the new data representation and the prior anomaly score corresponding to the first sample data to obtain a final abnormal data detection model.

[0029] A second aspect of an embodiment of the present invention discloses an abnormal data detection method, which is applicable to an abnormal data detection model constructed by the method disclosed in the first aspect of an embodiment of the present invention, comprising:

[0030] Determining the abnormal data detection model;

[0031] Acquire multiple data to be detected;

[0032] The multiple data to be detected are input into the abnormal data detection model for detection, and abnormal data in the multiple data to be detected are detected.

[0033] A third aspect of an embodiment of the present invention discloses a system for constructing an abnormal data detection model, including:

[0034] A first determining unit, configured to determine a plurality of first sample data and a plurality of second sample data from a dedicated data set for anomaly detection, wherein the first sample data are unlabeled data, and the second sample data are abnormal data carrying abnormal labels;

[0035] A clustering unit, used for clustering the second sample data to obtain at least one abnormal data cluster center;

[0036] an a priori estimation unit, configured to perform a priori estimation on the distribution of the plurality of first sample data to obtain an isolation score corresponding to each of the first sample data;

[0037] a first calculation unit, configured to calculate the similarity between the first sample data and the second sample data according to the distance between the first sample data and each of the abnormal data cluster centers, so as to obtain a distance score for the first sample data;

[0038] a second calculation unit, configured to calculate a priori anomaly score corresponding to the first sample data by using the isolation score and the distance score corresponding to the first sample data;

[0039] A second determining unit, configured to determine a priori anomaly score average and a priori anomaly score standard deviation based on a priori anomaly scores corresponding to a plurality of the first sample data;

[0040] A training unit is used to use the prior anomaly score mean and the prior anomaly score standard deviation as partial hyperparameters of a preset neural network model, and train the neural network model using the first sample data, the prior anomaly score corresponding to the first sample data, and the second sample data to obtain a final abnormal data detection model.

[0041] Preferably, the first determining unit includes:

[0042] A first extraction module is used to extract a preset percentage of data from a special data set for anomaly detection to obtain a model training set, wherein the special data set for anomaly detection includes normal data with normal labels and abnormal data with abnormal labels;

[0043] A determination module, used to determine the amount of noise according to the amount of normal data in the model training set;

[0044] A processing module, configured to randomly extract abnormal data from the model training set based on the amount of noise, and randomly divide the randomly extracted abnormal data into a plurality of groups of abnormal data, each group of abnormal data comprising two abnormal data;

[0045] A feature exchange module is used to exchange data features of preset dimensions for each group of divided abnormal data to obtain noise data;

[0046] A mixing module, used for randomly mixing the noise data with the normal data in the model training set to obtain a plurality of first sample data;

[0047] The second extraction module is used to randomly extract a plurality of abnormal data as second sample data from the model training set.

[0048] Preferably, the a priori estimation unit is specifically used to: use an isolation forest algorithm to perform a priori estimation on the distribution of multiple first sample data to obtain an isolation score corresponding to each first sample data.

[0049] A fourth aspect of an embodiment of the present invention discloses an abnormal data detection system, including:

[0050] A determination unit, used to determine an abnormal data detection model constructed by the method disclosed in the first aspect of the embodiment of the present invention;

[0051] An acquisition unit, used for acquiring a plurality of data to be detected;

[0052] The detection unit is used to input the multiple data to be detected into the abnormal data detection model for detection, and detect the abnormal data in the multiple data to be detected.

[0053] Based on the above-mentioned embodiment of the present invention, a method for constructing an abnormal data detection model, an abnormal data detection method and a system are provided. The construction method is: determining multiple first sample data and multiple second sample data from a special data set for abnormal detection, wherein the first sample data is unlabeled data and the second sample data is abnormal data carrying abnormal labels; clustering the second sample data to obtain at least one abnormal data cluster center; performing a priori estimation on the distribution of the multiple first sample data to obtain an isolation score corresponding to each first sample data; calculating the similarity between the first sample data and the second sample data according to the distance between the first sample data and each abnormal data cluster center to obtain a distance score of the first sample data; calculating the a priori abnormal score corresponding to the first sample data using the isolation score and the distance score corresponding to the first sample data; determining the a priori abnormal score average value and the a priori abnormal score standard deviation based on the a priori abnormal score corresponding to the multiple first sample data; using the a priori abnormal score average value and the a priori abnormal score standard deviation as part of the hyperparameters of the preset neural network model, and training the neural network model using the first sample data, the a priori abnormal score corresponding to the first sample data and the second sample data to obtain a final abnormal data detection model. This solution uses unlabeled data and abnormal data with abnormal labels to train the neural network model to obtain the final abnormal data detection model. The abnormal data in the massive data can be detected by the anomaly detection model to improve the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0055] Figure 1 A flowchart of a method for constructing an abnormal data detection model provided by an embodiment of the present invention;

[0056] Figure 2A schematic diagram of a method for constructing an abnormal data detection model provided by an embodiment of the present invention;

[0057] Figure 3 A flowchart of an abnormal data detection model obtained by training provided by an embodiment of the present invention;

[0058] Figure 4 A flowchart for determining a plurality of first sample data and a plurality of second sample data provided by an embodiment of the present invention;

[0059] Figure 5 A flowchart of an abnormal data detection method provided by an embodiment of the present invention;

[0060] Figure 6 A structural block diagram of a system for building an abnormal data detection model provided by an embodiment of the present invention;

[0061] Figure 7 A structural block diagram of an abnormal data detection system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0064] As can be seen from the background technology, due to reasons such as data dimensionality disaster and high computational complexity, traditional abnormal data detection methods cannot accurately detect abnormal data from high-dimensional data sets and massive data, and the detection accuracy is poor.

[0065] Therefore, the embodiment of the present invention provides a method for constructing an abnormal data detection model, an abnormal data detection method and a system, which trains a neural network model using unlabeled data and abnormal data with abnormal labels to obtain a final abnormal data detection model. The abnormal data in the massive data is detected by the abnormal detection model to improve the detection accuracy.

[0066] See also Figure 1 , shows a method for constructing an abnormal data detection model provided by an embodiment of the present invention, the construction method comprising:

[0067] Step S101: determining a plurality of first sample data and a plurality of second sample data from a data set dedicated to abnormality detection.

[0068] It should be noted that the first sample data is unlabeled data, and the second sample data is abnormal data with abnormal labels. Since the data in the special data set for anomaly detection are all labeled, that is, the special data set for anomaly detection contains normal data with normal labels and abnormal data with abnormal labels, in order to determine the unlabeled data (that is, the first sample data) from the special data set for anomaly detection, the abnormal data and normal data in the special data set for anomaly detection can be used to construct the unlabeled data.

[0069] In the specific implementation of step S101, noise data is generated based on normal data (which may be part) and abnormal data (which may be part) in the special data set for abnormality detection; an unlabeled data set with noise pollution is constructed by using the noise data and the normal data in the special data set for abnormality detection, and the data contained in the unlabeled data set with noise pollution can be used as unlabeled data (that is, first sample data). Abnormal data is randomly extracted from the special data set for abnormality detection as second sample data.

[0070] In some embodiments, the number of second sample data is extremely small relative to the number of first sample data, or in other words, the number of first sample data is much larger than the number of second sample data; for example: hundreds of thousands of first sample data are determined from the anomaly detection dedicated data set, and 5 to 60 second sample data are determined from the anomaly detection dedicated data set.

[0071] Step S102: clustering the second sample data to obtain at least one abnormal data cluster center.

[0072] In the specific implementation of step S102 , a clustering algorithm is used to cluster the determined plurality of second sample data (abnormal data) to obtain at least one abnormal data cluster center corresponding to the plurality of second sample data.

[0073] In some embodiments, the DBSCAN algorithm is used to cluster the plurality of second sample data to obtain a cluster center set C of abnormal data = {c 1 ,c 2 ,…,c q}, the cluster center set includes at least one abnormal data cluster center corresponding to multiple second sample data.

[0074] Step S103: performing a priori estimation on the distribution of the plurality of first sample data to obtain an isolation score corresponding to each first sample data.

[0075] In the specific implementation of step S103, the isolation forest algorithm is used to perform a priori estimation on the distribution of the plurality of first sample data to obtain an isolation score corresponding to each first sample data. Specifically, the isolation forest algorithm is used to estimate the isolation degree of each first sample data to calculate the isolation score of the first sample data.

[0076] It should be noted that the isolation degree of a certain first sample data can represent the difference between the first sample data and other first sample data; the isolation score of a certain first sample data can represent whether the first sample data is easy to be isolated (or isolated).

[0077] It can be seen from the content of the above step S101 that the first sample data is data in an unlabeled data set with noise pollution; in some embodiments, the distribution of data in the unlabeled data set U is estimated using the isolation forest algorithm to calculate the first sample data x in the unlabeled data set U. u The isolation fraction IS(x u ).

[0078] Step S104: Calculate the similarity between the first sample data and the second sample data according to the distance between the first sample data and each abnormal data cluster center to obtain a distance score of the first sample data.

[0079] In the specific process of implementing step S104, for each first sample data, the abnormal data cluster center closest to the first sample data is determined according to the distance between the first sample data and each abnormal data cluster center.

[0080] In some embodiments, the specific method of determining the distance between a first sample data and a cluster center of abnormal data is: using formula (1) to calculate the first sample data x in the unlabeled data set U u and abnormal data cluster center c j The Euclidean distance λ(x u ,c j ), in formula (1), d is the first sample data x u Dimension.

[0081]

[0082] By using the above formula (1), the distance between the first sample data and each abnormal data cluster center can be calculated; for each first sample data, the minimum distance between the first sample data and each abnormal data cluster center is determined, and the abnormal data cluster center corresponding to the minimum distance is the abnormal data cluster center closest to the first sample data.

[0083] In some embodiments, for the first sample data x u , determine x by formula (2) u The minimum distance to the center of all abnormal data clusters.

[0084]

[0085] For each first sample data, the similarity between the first sample data and the second sample data is calculated using the distance between the first sample data and the abnormal data cluster center closest to the first sample data (that is, the minimum distance mentioned above) to obtain the distance score of the first sample data.

[0086] In some embodiments, for the first sample data x u , calculate x by formula (3) u The similarity with the second sample data to get x u The distance score DS(x u ).

[0087]

[0088] Through the above formulas (1) to (3), the distance score of each first sample data is calculated. The distance score of the first sample data can represent the similarity between the first sample data and the abnormal data. The closer the distance score is to 1, the more likely the first sample data is to be potential abnormal data. The closer the distance score is to 0, the more likely the first sample data is to be normal data. In other words, the distance score can be used to measure the degree of abnormality of the first sample data.

[0089] It should be noted that the execution order of the above steps S102 to S104 is only an example; step S102 and step S104 may be executed first, and then step S103; or step S102 (step S104 is executed after step S102) and step S103 may be executed at the same time; the execution order of steps S102 to S104 is not limited here.

[0090] Step S105: Calculate a priori anomaly scores corresponding to the first sample data using the isolation scores and distance scores corresponding to the first sample data.

[0091] In the specific process of implementing step S105 , for each first sample data, the isolation score and the distance score corresponding to the first sample data are used to calculate the prior anomaly score corresponding to the first sample data.

[0092] Specifically, for the first sample data x u , according to the first sample data x u The distance score DS(x u ) and the isolation fraction IS(x u ), the first sample data x is calculated by formula (4) u The prior anomaly score p of .

[0093] p = (1-τ)DS(x u )+τIS(x u ) (4)

[0094] In formula (4), τ is the proportionality coefficient, τ∈(0,1) and p∈(0,1). The closer the value of p is to 1, the higher the probability that the first sample data x u The more likely it is that the data is a potential anomaly, the closer the value of p is to 0, indicating that the first sample data x u The more likely it is normal data.

[0095] It should be noted that in the subsequent step S107, during the training of the neural network model, the first sample data x u The prior anomaly score p will constitute the loss function x u training weights, thereby reducing the impact of noise data in the training process.

[0096] By the above method, the prior anomaly scores corresponding to all the first sample data in the unlabeled dataset U are calculated. The prior anomaly scores corresponding to all the first sample data can be recorded as P = {p 1 ,…,p n}.

[0097] Step S106: Based on the prior anomaly scores corresponding to the plurality of first sample data, determine the prior anomaly score average and the prior anomaly score standard deviation.

[0098] In the process of specifically implementing step S106, in order to fully mine the information contained in all the first sample data in the unlabeled data set U, after calculating the prior anomaly scores corresponding to all the first sample data, the prior anomaly score average and the prior anomaly score standard deviation are determined based on the prior anomaly scores corresponding to multiple first sample data; the prior anomaly score average is: the average (or expectation) of the prior anomaly scores of all the first sample data in the unlabeled data set U; the prior anomaly score standard deviation is: the standard deviation of the prior anomaly scores of all the first sample data in the unlabeled data set U.

[0099] Specifically, we first calculate the average value of the prior anomaly score μ by formula (5): r , and then calculate the prior anomaly score standard deviation σ through formula (6) r .

[0100]

[0101]

[0102] Step S107: using the mean value of the prior anomaly score and the standard deviation of the prior anomaly score as part of the hyperparameters of the preset neural network model, and training the neural network model using the first sample data, the prior anomaly score corresponding to the first sample data, and the second sample data to obtain a final anomaly data detection model.

[0103] It should be noted that the prior anomaly score mean value and the prior anomaly score standard deviation are used as some hyperparameters of the preset neural network model, and the prior anomaly score mean value and the prior anomaly score standard deviation can be used to guide the training of the neural network model.

[0104] In the process of implementing step S107, data dimension reduction and feature extraction are performed on each first sample data and each second sample data to obtain a new data representation; specifically, the automatic feature extraction capability of the neural network model is utilized to perform data dimension reduction and feature extraction on the first sample data to obtain a new data representation, and to perform data dimension reduction and feature extraction on the second sample data to obtain a new data representation.

[0105] The prior anomaly score average and the prior anomaly score standard deviation are used as part of the hyperparameters of the preset neural network model, and the neural network model is trained using the new data representation and the prior anomaly scores corresponding to each first sample data to obtain the final abnormal data detection model. Specifically, the prior anomaly score average and the prior anomaly score standard deviation are input into the neural network model as hyperparameters, and the neural network model is trained using the new data representation and the prior anomaly scores corresponding to each first sample data until the neural network model converges to obtain the final abnormal data detection model (or abnormal data detector), and the loss function used in the training process is the preset loss function.

[0106] It should be noted that, during the training process of the neural network model mentioned above, a new data representation of a certain sample data (first sample data or second sample data) is input into the neural network model for scoring, and the abnormal score of the sample data can be output.

[0107] It should be noted that the first sample data and the second sample data used above are part of the data in the special data set for anomaly detection. In some embodiments, a test data set can be obtained from the special data set for anomaly detection. The data in the test data set is: other data in the special data set for anomaly detection other than the data used to determine the first sample data and the second sample data. Preferably, the test data set is used to test the effect of the anomaly data detection model, and the parameters of the anomaly data detection model are adjusted according to the test effect.

[0108] In some embodiments, the architecture of the neural network model used to train the abnormal data detection model is explained through the following content. The neural network model mentioned in the following content is: the neural network model used to train the abnormal data detection model.

[0109] An explanation of the architecture of the neural network model and the loss function.

[0110] In the input space of the neural network model, And the output space is In the case of, let the representation space after the original data feature extraction be Among them, d and q represent the dimensions of the data samples, represents a vector of dimension d, Represents a vector of dimension q.

[0111] The objective function of the neural network model Divided into feature extraction functions and anomaly scoring function And k = φ(·; θ f ) and θ={θ f ,θ s}; where ψ(·; θ) is the objective function, φ(·; θ f ) is the feature extraction function, is the anomaly scoring function, θ f is the parameter of the neural network (which can be called the feature extraction layer) used to extract new data representation, θ s are the parameters of the neural network (which can be called the anomaly scoring layer) used to calculate the anomaly score of the sample data.

[0112] The objective function of the feature extraction layer is φ(·; θ f ), the feature extraction layer is a multi-layer fully connected neural network, and the output of the feature extraction layer is the new data representation k∈κ.

[0113] It should be noted that and φ(·;θ f ) is the same function, In terms of expression, φ(·;θf ) More mapping relationships, The objective function φ(·;θ f )’s input space and output space.

[0114] In some embodiments, θ f = {W 1 , W 2 , …, W L}, where W l is the parameter of the lth hidden layer; the output of the lth hidden layer is and In the case of x l The specific content is as follows: l is the output of the lth hidden layer.

[0115]

[0116] in, The representative dimension is The vector space of , η(·) is the ReLU function.

[0117] From the above content, it can be seen that the new data representation of a certain sample data is input into the neural network model for scoring, and the abnormal score of the sample data can be output; in some embodiments, the abnormal scoring layer can use a single-layer neural network with a linear activation function for scoring, and the objective function of the abnormal scoring layer is The input is the new data representation k, and the output is the anomaly score s of the sample data.

[0118] The parameters θ in the anomaly scoring layer s ={ω 1 ,…,ω q , b}, the neural network model can be determined, and the specific content of the neural network model is shown in formula (8).

[0119]

[0120] In formula (8), ω is the weight term in the parameters of the anomaly scoring layer, and b is the bias term in the parameters of the anomaly scoring layer.

[0121] The above is a relevant explanation about the architecture of the neural network model. In the process of training the neural network model, the loss function used can be a concentration loss function. Specifically, other loss functions can be selected according to actual conditions. In the embodiment of the present invention, there is no limitation on the loss function for training the neural network model.

[0122] In some embodiments, the concentration loss function is specifically as shown in formula (9).

[0123] L = (1-α)(1-y)(1-p) β |gap(x)|+αymax(0,m-gap(x)) (9)

[0124] In formula (9), when sample data x∈A, y=1, where A is a data set including first sample data and second sample data; ψ(x) is the prior anomaly score corresponding to the data sample x, μ r is the mean of the prior anomaly scores, σ r is the standard deviation of the prior anomaly score; α∈[0,1] is a class balance parameter used to solve the imbalance of positive and negative sample data (first sample data and second sample data), thereby increasing the proportion of the second sample data in the loss function; β is the model robustness parameter; (1-p) β It is used to alleviate the negative impact of noise data in the unlabeled dataset U on model accuracy.

[0125] It should be noted that the a priori anomaly score p mentioned in the above step S105 will constitute the x in the loss function. u The training weight of , (1-p) in formula (9) β That is x u The training weights of .

[0126] In combination with the above formula (9), the content of "the prior anomaly score average and the prior anomaly score standard deviation can be used to guide the training of the neural network model" in the above step S107 is explained.

[0127] Let x a ∈A and x u ∈U, minimize the concentration loss function, the first sample data x u The anomaly score ψ(x u ) will be as close as possible to the prior anomaly score average μ r , the second sample data x a The anomaly score ψ(x a ) and μ r There is at least mσ between r The difference between |ψ(x a )-μ r |≥mσ r , where m is a hyperparameter that makes the abnormal scores of positive and negative sample data have significant deviations, σ r It is the standard deviation of the prior anomaly score. It is worth noting that if the optimization starts with x a The anomaly score ψ(x a)<0, the Loss of the neural network model will be too large. Therefore, in the process of optimizing the neural network model, x a Assign a large positive anomaly score. Since the vast majority of sample data in the unlabeled dataset U is normal data, and μ r It can be regarded as the expected estimate of the prior anomaly score p of normal data, so the converged neural network model can make the abnormal data x 1 and normal data x 2 The anomaly score has a significant deviation, that is, the objective function ψ(x;θ) satisfies ψ(x 1 )>ψ(x 2 ).

[0128] The above is a description of the architecture and loss function of the neural network model. It can be seen from the contents of the above steps that in the weakly supervised abnormal data detection method, the embodiment of the present invention can use a large amount of first sample data (unlabeled) and a small amount of second sample data (labeled) to train an abnormal data detection model that can be used to identify abnormal data; compared with the traditional weakly supervised abnormal data detection method, the embodiment of the present invention does not need to spend a high manpower cost to label the data, which can save costs. In addition, the abnormal data in the massive data is detected by the abnormal detection model to improve the detection accuracy.

[0129] To better explain the above embodiments of the present invention Figure 1 The content in Figure 2 The principle architecture diagram of the method for constructing the abnormal data detection model is explained in detail.

[0130] Figure 2 In the schematic diagram of the construction method of the normal data detection model shown, the construction method is mainly divided into a priori estimation part 100 and an abnormal scoring network part 200, wherein: Figure 2 The abnormal scoring network in is the neural network model mentioned in the above embodiment.

[0131] The specific content of the prior estimation part 100 is: clustering a very small number of labeled anomalies (that is, the second sample data) to obtain an anomaly cluster center (that is, an abnormal data cluster center). The distance score of the unlabeled data is calculated using the anomaly cluster center and the large-scale unlabeled data (that is, the first sample data), and the isolation score of the unlabeled data is calculated using the isolation forest algorithm. The a priori anomaly score is calculated using the distance score and isolation score of the unlabeled data. The very small number of labeled anomalies, the large-scale unlabeled data, and the a priori anomaly score are used as inputs to the anomaly scoring network part 200.

[0132] The specific content of the anomaly scoring network part 200 is: based on a very small amount of labeled anomalies from the prior estimation part 100, large-scale unlabeled data and prior anomaly scores, the neural network model is trained until the neural network model converges to obtain the final abnormal data detection model.

[0133] To better explain the above embodiments of the present invention Figure 1 The process of training the neural network model involved in step S107 is performed by Figure 3 An example is given below. It should be noted that Figure 3 In the example of training the neural network model shown, the RMSProp gradient descent algorithm is used for model training. b / 2 sample data are extracted from the unlabeled data set (including the first sample data) and the labeled abnormal data set (including the second sample data) as mini-batches. The detailed training process is shown below. Figure 3 The content in.

[0134] See also Figure 3 , shows a flow chart of training an abnormal data detection model provided by an embodiment of the present invention, including the following steps:

[0135] Step S301: Set i=1 and j=1.

[0136] Step S302: Determine whether i is less than or equal to epochs. If i is less than or equal to epochs, execute step S303; if i is greater than epochs, it means that the neural network model has converged, and return to the abnormal data detection model.

[0137] It should be noted that epochs is the total number of rounds for training the neural network model.

[0138] Step S303: Determine whether j is less than or equal to batches. If j is less than or equal to batches, execute step S304; if j is greater than batches, reset j to 1, and i=i+1, and return to execute step S302.

[0139] It should be noted that the size of batches represents the number of mini-batches randomly collected in each round of training, or in other words, batches = b.

[0140] Step S304: randomly extract b / 2 first sample data from the unlabeled data set, and randomly extract b / 2 second sample data from the labeled abnormal data set.

[0141] Step S305: Input the randomly selected first sample data and the second sample data into the neural network model to calculate the anomaly score of the first sample data and the anomaly score of the second sample data.

[0142] Step S306: input the anomaly score of the first sample data, the anomaly score corresponding to the second sample data, and the a priori anomaly score of the randomly selected first sample data into a concentration loss function for calculation.

[0143] Step S307: According to the calculation result of the concentration loss function, execute the RMSProp gradient descent algorithm and update the parameters of the neural network model. Return to step S303.

[0144] In the process of implementing step S307, the calculation result obtained by processing the anomaly score of the first sample data, the anomaly score corresponding to the second sample data and the prior anomaly score of the randomly selected first sample data according to the concentration loss function is executed, and the parameters of the neural network model are updated, j=j+1, and the execution of step S303 is returned.

[0145] The above is about training the neural network model to obtain the abnormal data detection model.

[0146] The above embodiments of the present invention Figure 1 For the process of determining a plurality of first sample data and a plurality of second sample data involved in step S101, see Figure 4 , shows a flowchart of determining a plurality of first sample data and a plurality of second sample data provided by an embodiment of the present invention, including the following steps:

[0147] Step S401: extracting a preset percentage of data from a special data set for anomaly detection to obtain a model training set.

[0148] It should be noted that the anomaly detection dedicated data set contains normal data with normal labels and abnormal data with abnormal labels.

[0149] In the specific implementation of step S401, a preset percentage of data in the special data set for anomaly detection is extracted to obtain a model training set, and other data in the special data set for anomaly detection except the model training set can be used as a test data set.

[0150] For example, 80% of the data is extracted from the anomaly detection dedicated data set as the model training set, and the other 20% of the data in the anomaly detection dedicated data set except the model training set is used as the test data set.

[0151] Step S402: Determine the amount of noise according to the amount of normal data in the model training set.

[0152] In the specific implementation of step S402, the preset noise pollution rate can be used in combination with the amount of normal data in the model training set to calculate the amount of noise that needs to be added to the model training set.

[0153] Step S403: based on the amount of noise, randomly extract abnormal data from the model training set, and randomly divide the randomly extracted abnormal data into multiple groups of abnormal data.

[0154] In the specific implementation of step S403, based on the amount of noise, abnormal data in the model training set is randomly extracted; specifically, the number of noise abnormal data is randomly extracted from the model training set.

[0155] The randomly selected abnormal data are randomly divided into multiple groups of abnormal data, each group of abnormal data contains two abnormal data. In other words, any two abnormal data are randomly obtained as a group of abnormal data.

[0156] Step S404: performing data feature exchange of preset dimensions on each group of divided abnormal data to obtain noise data.

[0157] In the specific implementation of step S404, data features of preset dimensions are exchanged for each group of divided abnormal data to obtain noise data. Specifically, for each group of abnormal data, two abnormal data in the group of abnormal data are exchanged for data features of preset dimensions to obtain two noise data.

[0158] For example: for abnormal data A and abnormal data B in a certain group of abnormal data, randomly extract the data features of 5% dimension of abnormal data A, and randomly extract the data features of 5% dimension of abnormal data B; exchange the data features of 5% dimension of abnormal data A with the data features of 5% dimension of abnormal data B, and two noise data can be obtained.

[0159] In summary, the contents of step S403 and step S404, that is, for the abnormal data randomly extracted from the model training set, the data features of the abnormal data are randomly exchanged between two of them to construct noise data.

[0160] Step S405: Randomly mix the noise data with the normal data in the model training set to obtain a plurality of first sample data.

[0161] In the process of implementing step S405, normal data in the model training set is obtained; the constructed noise data is randomly mixed with the obtained normal data to construct an unlabeled data set with noise pollution; the data in the unlabeled data set with noise pollution can be used as unlabeled data (first sample data).

[0162] Step S406: randomly extracting a plurality of abnormal data as second sample data from the model training set.

[0163] In the specific implementation of step S406, a plurality of abnormal data (a small amount of abnormal data) are randomly extracted from the model training set, and the extracted abnormal data are the second sample data.

[0164] It should be noted that the order of determining the first sample data and determining the second sample data shown in the above steps is only for illustrative purposes. The first sample data may be determined first and then the second sample data, or the second sample data may be determined first and then the first sample data. The order of determining the first sample data and determining the second sample data is not limited herein.

[0165] The above embodiments are related explanations on building an abnormal data detection model, using unlabeled data and abnormal data with abnormal labels to train a neural network model to obtain the final abnormal data detection model. The abnormal data in the massive data is detected by the abnormal detection model to improve the detection accuracy.

[0166] Corresponding to the method for constructing an abnormal data detection model provided in the above embodiment of the present invention, see Figure 5 The embodiment of the present invention also provides a flowchart of an abnormal data detection method, comprising the following steps:

[0167] Step S501: Determine an abnormal data detection model.

[0168] In the process of specifically implementing step S501, an abnormal data detection model is determined. The abnormal data detection model is constructed based on the abnormal data detection model construction method disclosed in the above method embodiment.

[0169] Step S502: Acquire a plurality of data to be detected.

[0170] In the specific process of implementing step S502, a plurality of to-be-detected data for which abnormalities need to be detected are obtained.

[0171] Step S503: inputting a plurality of data to be detected into an abnormal data detection model for detection, and detecting abnormal data from the plurality of data to be detected.

[0172] In the process of specifically implementing step S503, multiple data to be detected are input into the abnormal data detection model for detection to obtain abnormal scores of the multiple data to be detected; according to the abnormal scores of each data to be detected, abnormal data among the multiple data to be detected is determined; specifically, the data to be detected whose abnormal score is greater than the abnormal score threshold is determined to be abnormal data.

[0173] In the embodiment of the present invention, an abnormal data detection model is determined and a plurality of data to be detected is obtained, and the abnormal data in the plurality of data to be detected is detected by the abnormal data detection model to improve the detection accuracy.

[0174] Corresponding to the method for constructing an abnormal data detection model provided in the above embodiment of the present invention, see Figure 6 , the embodiment of the present invention also provides a structural block diagram of a construction system for an abnormal data detection model, the construction system comprising: a first determination unit 601, a clustering unit 602, a priori estimation unit 603, a first calculation unit 604, a second calculation unit 605, a second determination unit 606 and a training unit 607;

[0175] The first determining unit 601 is used to determine a plurality of first sample data and a plurality of second sample data from the anomaly detection dedicated data set, wherein the first sample data is unlabeled data, and the second sample data is anomaly data carrying anomaly labels.

[0176] The clustering unit 602 is used to cluster the second sample data to obtain at least one abnormal data cluster center.

[0177] The prior estimation unit 603 is used to perform a priori estimation on the distribution of the plurality of first sample data to obtain an isolation score corresponding to each first sample data.

[0178] In a specific implementation, the prior estimation unit 603 is specifically used to: use the isolation forest algorithm to perform a priori estimation on the distribution of multiple first sample data to obtain an isolation score corresponding to each first sample data.

[0179] The first calculation unit 604 is used to calculate the similarity between the first sample data and the second sample data according to the distance between the first sample data and each abnormal data cluster center, so as to obtain the distance score of the first sample data.

[0180] In a specific implementation, the first calculation unit 604 is specifically used to: for each first sample data, determine the abnormal data cluster center closest to the first sample data based on the distance between the first sample data and each abnormal data cluster center; for each first sample data, use the distance between the first sample data and the abnormal data cluster center closest to the first sample data to calculate the similarity between the first sample data and the second sample data to obtain the distance score of the first sample data.

[0181] The second calculation unit 605 is used to calculate the prior anomaly score corresponding to the first sample data by using the isolation score and the distance score corresponding to the first sample data.

[0182] The second determining unit 606 is used to determine the prior anomaly score average and the prior anomaly score standard deviation based on the prior anomaly scores corresponding to the plurality of first sample data.

[0183] The training unit 607 is used to use the mean value of the prior anomaly score and the standard deviation of the prior anomaly score as part of the hyperparameters of the preset neural network model, and train the neural network model using the first sample data, the prior anomaly score corresponding to the first sample data, and the second sample data to obtain a final anomaly data detection model.

[0184] In a specific implementation, the training unit 607 is specifically used to: perform data dimensionality reduction and feature extraction on the first sample data and the second sample data to obtain a new data representation; use the mean value of the prior anomaly score and the standard deviation of the prior anomaly score as part of the hyperparameters of the preset neural network model, and use the new data representation and the prior anomaly score corresponding to the first sample data to train the neural network model to obtain the final abnormal data detection model.

[0185] In the embodiment of the present invention, the neural network model is trained using unlabeled data and abnormal data with abnormal labels to obtain a final abnormal data detection model. The abnormal data in the massive data is detected by the abnormal detection model to improve the detection accuracy.

[0186] Preferably, combined Figure 6 As shown in the figure, the first determination unit 601 includes: a first extraction module, a determination module, a processing module, a feature exchange module, a mixing module and a second extraction module; the execution principle of each module is as follows:

[0187] A first extraction module is used to extract a preset percentage of data from a special data set for anomaly detection to obtain a model training set, wherein the special data set for anomaly detection includes normal data with normal labels and abnormal data with abnormal labels;

[0188] The determination module is used to determine the amount of noise according to the amount of normal data in the model training set.

[0189] The processing module is used for randomly extracting abnormal data from the model training set based on the amount of noise, and randomly dividing the randomly extracted abnormal data into multiple groups of abnormal data, each group of abnormal data contains two abnormal data.

[0190] The feature exchange module is used to exchange data features of preset dimensions for each group of divided abnormal data to obtain noise data.

[0191] The mixing module is used to randomly mix the noise data with the normal data in the model training set to obtain multiple first sample data.

[0192] The second extraction module is used to randomly extract a plurality of abnormal data as second sample data from the model training set.

[0193] Corresponding to the abnormal data detection method provided by the above embodiment of the present invention, see Figure 7 , the embodiment of the present invention also provides a structural block diagram of an abnormal data detection system, the abnormal data detection system includes: a determination unit 701, an acquisition unit 702 and a detection unit 703;

[0194] The determination unit 701 is used to determine the abnormal data detection model constructed by the abnormal data detection model construction method disclosed in the above method embodiment.

[0195] The acquisition unit 702 is used to acquire a plurality of to-be-detected data.

[0196] The detection unit 703 is used to input the multiple data to be detected into the abnormal data detection model for detection, and detect the abnormal data in the multiple data to be detected.

[0197] In the embodiment of the present invention, an abnormal data detection model is determined and a plurality of data to be detected is obtained, and the abnormal data in the plurality of data to be detected is detected by the abnormal data detection model to improve the detection accuracy.

[0198] In summary, the embodiments of the present invention provide a method for constructing an abnormal data detection model, an abnormal data detection method and a system, which can train an abnormal data detection model that can be used to identify abnormal data using a large amount of first sample data (unlabeled) and a small amount of second sample data (labeled); compared with the traditional weakly supervised abnormal data detection method, the embodiments of the present invention do not need to spend a high manpower cost to label data, which can save costs. In addition, the abnormal data in the massive data is detected by the abnormal detection model to improve the detection accuracy.

[0199] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0200] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0201] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing an abnormal data detection model, characterized in that: Applied to network attack detection and credit card abnormal transaction detection, the method includes: Determine a plurality of first sample data and a plurality of second sample data from a dedicated anomaly detection data set, wherein the first sample data is unlabeled data, and the second sample data is abnormal data carrying an abnormality label; Clustering the second sample data to obtain at least one abnormal data cluster center; Performing a priori estimation on the distribution of the plurality of first sample data to obtain an isolation score corresponding to each of the first sample data; Calculating the similarity between the first sample data and the second sample data according to the distance between the first sample data and each of the abnormal data cluster centers to obtain a distance score for the first sample data; Calculating a priori anomaly scores corresponding to the first sample data using the isolation scores and the distance scores corresponding to the first sample data; Determine a priori anomaly score average and a priori anomaly score standard deviation based on a plurality of prior anomaly scores corresponding to the first sample data; The prior anomaly score average and the prior anomaly score standard deviation are used as part of the hyperparameters of the preset neural network model, and the neural network model is trained using the first sample data, the prior anomaly score corresponding to the first sample data, and the second sample data to obtain a final abnormal data detection model; wherein the abnormal data detection model is used to detect multiple data to be detected to detect abnormal data in the multiple data to be detected; The method of using the prior anomaly score average and the prior anomaly score standard deviation as partial hyperparameters of a preset neural network model, and using the first sample data, the prior anomaly score corresponding to the first sample data, and the second sample data to train the neural network model to obtain a final abnormal data detection model includes: performing data dimension reduction and feature extraction on the first sample data and the second sample data to obtain a new data representation; using the prior anomaly score average and the prior anomaly score standard deviation as partial hyperparameters of a preset neural network model, and using the new data representation and the prior anomaly score corresponding to the first sample data to train the neural network model to obtain a final abnormal data detection model.

2. The method according to claim 1, characterized in that: Determining a plurality of first sample data and a plurality of second sample data from a dedicated anomaly detection data set includes: Extracting a preset percentage of data from a special data set for anomaly detection to obtain a model training set, wherein the special data set for anomaly detection includes normal data with normal labels and abnormal data with abnormal labels; Determining the amount of noise according to the amount of normal data in the model training set; Based on the amount of noise, randomly extract abnormal data from the model training set, and randomly divide the randomly extracted abnormal data into multiple groups of abnormal data, each group of abnormal data includes two abnormal data; Perform data feature swapping of preset dimensions on each group of abnormal data to obtain noise data; Randomly mixing the noise data with normal data in the model training set to obtain a plurality of first sample data; A plurality of abnormal data as second sample data are randomly extracted from the model training set.

3. The method according to claim 1, characterized in that A priori estimation is performed on the distribution of the plurality of first sample data to obtain an isolation score corresponding to each of the first sample data, including: An isolation forest algorithm is used to perform a priori estimation on the distribution of the plurality of first sample data to obtain an isolation score corresponding to each of the first sample data.

4. The method according to claim 1, characterized in that: Calculating the similarity between the first sample data and the second sample data according to the distance between the first sample data and each of the abnormal data cluster centers to obtain a distance score of the first sample data includes: For each first sample data, determining the abnormal data cluster center closest to the first sample data according to the distance between the first sample data and each abnormal data cluster center; For each first sample data, the similarity between the first sample data and the second sample data is calculated using the distance between the first sample data and the abnormal data cluster center closest to the first sample data to obtain a distance score of the first sample data.

5. A method for detecting abnormal data, characterized in that: An abnormal data detection model suitable for being constructed by the method according to any one of claims 1 to 4, comprising: Determining the abnormal data detection model; Acquire multiple data to be detected; The multiple data to be detected are input into the abnormal data detection model for detection, and abnormal data in the multiple data to be detected are detected.

6. A system for constructing an abnormal data detection model, characterized in that: Applied to network attack detection and credit card abnormal transaction detection, including: A first determining unit, configured to determine a plurality of first sample data and a plurality of second sample data from a dedicated data set for anomaly detection, wherein the first sample data are unlabeled data, and the second sample data are abnormal data carrying abnormal labels; A clustering unit, used for clustering the second sample data to obtain at least one abnormal data cluster center; an a priori estimation unit, configured to perform a priori estimation on the distribution of the plurality of first sample data to obtain an isolation score corresponding to each of the first sample data; a first calculation unit, configured to calculate the similarity between the first sample data and the second sample data according to the distance between the first sample data and each of the abnormal data cluster centers, so as to obtain a distance score for the first sample data; a second calculation unit, configured to calculate a priori anomaly score corresponding to the first sample data by using the isolation score and the distance score corresponding to the first sample data; A second determining unit, configured to determine a priori anomaly score average and a priori anomaly score standard deviation based on a priori anomaly scores corresponding to a plurality of the first sample data; A training unit, used to use the prior anomaly score mean value and the prior anomaly score standard deviation as partial hyperparameters of a preset neural network model, and train the neural network model using the first sample data, the prior anomaly score corresponding to the first sample data, and the second sample data to obtain a final abnormal data detection model; wherein the abnormal data detection model is used to detect multiple data to be detected to detect abnormal data in the multiple data to be detected; The training unit is specifically used to perform data dimensionality reduction and feature extraction on the first sample data and the second sample data to obtain a new data representation; use the prior anomaly score mean and the prior anomaly score standard deviation as part of the hyperparameters of a preset neural network model, and train the neural network model using the new data representation and the prior anomaly score corresponding to the first sample data to obtain a final abnormal data detection model.

7. The system according to claim 6, characterized in that The first determining unit includes: A first extraction module is used to extract a preset percentage of data from a special data set for anomaly detection to obtain a model training set, wherein the special data set for anomaly detection includes normal data with normal labels and abnormal data with abnormal labels; A determination module, used to determine the amount of noise according to the amount of normal data in the model training set; A processing module, configured to randomly extract abnormal data from the model training set based on the amount of noise, and randomly divide the randomly extracted abnormal data into a plurality of groups of abnormal data, each group of abnormal data comprising two abnormal data; A feature exchange module is used to exchange data features of preset dimensions for each group of divided abnormal data to obtain noise data; A mixing module, used for randomly mixing the noise data with the normal data in the model training set to obtain a plurality of first sample data; The second extraction module is used to randomly extract a plurality of abnormal data as second sample data from the model training set.

8. The system according to claim 6, characterized in that The a priori estimation unit is specifically used to: use the isolation forest algorithm to perform a priori estimation on the distribution of multiple first sample data to obtain an isolation score corresponding to each first sample data.

9. An abnormal data detection system, characterized in that: include: A determination unit, configured to determine an abnormal data detection model constructed by the method according to any one of claims 1 to 4; An acquisition unit, used for acquiring a plurality of data to be detected; The detection unit is used to input the multiple data to be detected into the abnormal data detection model for detection, and detect the abnormal data in the multiple data to be detected.

Images