Abnormal data detection method, device, storage medium and computer equipment

Abstract

The present invention discloses a method, device, storage medium and computer equipment for detecting abnormal data, which relates to the field of information technology, and mainly aims to improve the detection efficiency and accuracy of abnormal data. The method comprises: obtaining attribute features corresponding to a data set to be detected; inputting the attribute features into a preset abnormal detection algorithm prediction model for algorithm prediction to obtain a target abnormal detection algorithm applicable to the data set to be detected; using the target abnormal detection algorithm to detect abnormal data in the data set to be detected, and obtaining abnormal data detection results corresponding to the data set to be detected. The present invention is suitable for detecting abnormal data.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method, device, storage medium and computer equipment for detecting abnormal data. Background Art

[0002] In the Internet age, the application of big data is everywhere. Faced with a wide variety of massive data in various industries, in order to ensure the availability of data and facilitate the analysis of subsequent data, it is necessary to detect abnormal data in the data set.

[0003] At present, abnormal data in a data set is usually detected by manually selecting anomaly detection algorithms. However, this method requires manual analysis of data features in advance, and the selection of anomaly detection algorithms based on the experience of the staff, and then the data is detected for anomalies based on the selected anomaly detection algorithm, which will result in low detection efficiency of abnormal data. At the same time, this method of manually selecting anomaly detection algorithms is greatly affected by the subjective factors of the staff, and there is no guarantee that the selected anomaly detection algorithm is suitable for the data set. If the selected anomaly detection algorithm is not suitable for the data set, the detection accuracy of abnormal data will be low. Summary of the invention

[0004] The present invention provides a method, device, storage medium and computer equipment for detecting abnormal data, which are mainly capable of improving the detection efficiency and detection accuracy of abnormal data.

[0005] According to a first aspect of the present invention, there is provided a method for detecting abnormal data, comprising:

[0006] Obtain the attribute features corresponding to the data set to be detected;

[0007] Inputting the attribute features into a preset anomaly detection algorithm prediction model for algorithm prediction to obtain a target anomaly detection algorithm applicable to the data set to be detected;

[0008] The target anomaly detection algorithm is used to detect the abnormal data in the data set to be detected, and an abnormal data detection result corresponding to the data set to be detected is obtained.

[0009] According to a second aspect of the present invention, there is provided a device for detecting abnormal data, comprising:

[0010] An acquisition unit, used to acquire attribute features corresponding to the data set to be detected;

[0011] An algorithm prediction unit is used to input the attribute features into a preset anomaly detection algorithm prediction model to perform algorithm prediction, so as to obtain a target anomaly detection algorithm applicable to the data set to be detected;

[0012] The anomaly detection unit is used to detect the abnormal data in the data set to be detected by using the target anomaly detection algorithm to obtain the abnormal data detection result corresponding to the data set to be detected.

[0013] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0014] Obtain the attribute features corresponding to the data set to be detected;

[0015] Inputting the attribute features into a preset anomaly detection algorithm prediction model for algorithm prediction to obtain a target anomaly detection algorithm applicable to the data set to be detected;

[0016] The target anomaly detection algorithm is used to detect the abnormal data in the data set to be detected, and an abnormal data detection result corresponding to the data set to be detected is obtained.

[0017] According to a fourth aspect of the present invention, there is provided a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the following steps are implemented:

[0018] Obtain the attribute features corresponding to the data set to be detected;

[0019] Inputting the attribute features into a preset anomaly detection algorithm prediction model for algorithm prediction to obtain a target anomaly detection algorithm applicable to the data set to be detected;

[0020] The target anomaly detection algorithm is used to detect the abnormal data in the data set to be detected, and an abnormal data detection result corresponding to the data set to be detected is obtained.

[0021] According to an abnormal data detection method, device, storage medium and computer equipment provided by the present invention, compared with the current method of detecting abnormal data in a data set by manually selecting an abnormal detection algorithm, the present invention obtains the attribute characteristics corresponding to the data set to be detected; and inputs the attribute characteristics into a preset abnormal detection algorithm prediction model for algorithm prediction to obtain a target abnormal detection algorithm applicable to the data set to be detected; finally, the abnormal data in the data set to be detected is detected using the target abnormal detection algorithm to obtain an abnormal data detection result corresponding to the data set to be detected, thereby using the preset abnormal detection algorithm prediction model to predict the abnormal detection algorithm applicable to the data set to be detected, and using the predicted abnormal detection algorithm to detect the abnormal data in the data set to be detected to obtain an abnormal detection result corresponding to the data set to be detected, thereby improving the efficiency of abnormal data detection. At the same time, this method can avoid the situation where an inappropriate abnormal detection algorithm is selected to detect the data set to be detected due to subjective factors of the staff, thereby improving the detection accuracy of abnormal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0023] Figure 1 A flow chart of a method for detecting abnormal data provided by an embodiment of the present invention is shown;

[0024] Figure 2 A flow chart of another abnormal data detection method provided by an embodiment of the present invention is shown;

[0025] Figure 3 A schematic diagram showing the structure of an abnormal data detection device provided by an embodiment of the present invention is shown;

[0026] Figure 4 A schematic diagram showing the structure of another abnormal data detection device provided by an embodiment of the present invention;

[0027] Figure 5 A schematic diagram of the physical structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0028] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0029] At present, the method of manually selecting anomaly detection algorithms to detect abnormal data in data sets results in low efficiency of abnormal data detection. At the same time, this method of manually selecting anomaly detection algorithms is greatly affected by subjective factors of staff, and there is no guarantee that the selected anomaly detection algorithm is suitable for the data set. If the selected anomaly detection algorithm is not suitable for the data set, it will lead to low detection accuracy of abnormal data.

[0030] In order to solve the above problems, an embodiment of the present invention provides a method for detecting abnormal data, such as Figure 1 As shown, the method includes:

[0031] 101. Obtain attribute features corresponding to the data set to be detected.

[0032] Among them, the data sets to be tested include the pyod public data set, the anonymized enterprise public data set, etc.

[0033] For the embodiments of the present invention, in order to overcome the defects of low efficiency and low accuracy in detecting abnormal data in the prior art, the embodiments of the present invention predict the abnormal detection algorithm applicable to the data set to be detected by using a preset abnormal detection algorithm prediction model, and detect the abnormal data in the data set to be detected by using the predicted abnormal detection algorithm to obtain the abnormal detection result corresponding to the data set to be detected, thereby improving the efficiency of abnormal data detection. At the same time, this method can avoid the situation where an inappropriate abnormal detection algorithm is selected to detect the data set to be detected due to subjective factors of the staff, thereby improving the detection accuracy of abnormal data. The embodiments of the present invention are mainly applied to the scene of detecting abnormal data. The execution subject of the embodiments of the present invention is a device or equipment capable of detecting abnormal data, which can be specifically set on the client or server side.

[0034] Specifically, relevant data sets can be collected in combination with specific business types and application scenarios. An embodiment of the present invention can collect a set of data to be detected on the Internet. At the same time, the enterprise database also contains a large number of different types of data sets. After the data sets in the enterprise database are desensitized, the data set to be detected can be obtained, and feature engineering processing is performed on the data set to be detected to obtain attribute features corresponding to the data set to be detected. Thereafter, the attribute features corresponding to the data set to be detected are input into a preset anomaly detection algorithm prediction model for algorithm prediction to obtain a target anomaly detection algorithm applicable to the data set to be detected. Finally, the target anomaly detection algorithm is used to detect the abnormal data in the data set to be detected to obtain the abnormal data detection result corresponding to the data set to be detected, thereby improving the detection efficiency of abnormal data in the data set to be detected.

[0035] 102. Input the attribute features into a preset anomaly detection algorithm prediction model to perform algorithm prediction, and obtain a target anomaly detection algorithm applicable to the data set to be detected.

[0036] Among them, the attribute characteristics include the mean, maximum value, minimum value, coefficient of fluctuation, standard deviation value, 25% percentile value, 50% percentile value, 75% percentile value, etc. corresponding to each data in the data set to be detected. The target anomaly detection algorithm includes linear model detection algorithm, neighbor-based detection algorithm, probability-based detection algorithm, integration-based detection algorithm and neural network detection algorithm. Among them, the linear detection algorithms include: PCA (principal component analysis algorithm), MCD (minimum covariance detection algorithm), OCSVM (single-class support vector machine algorithm), etc. The neighbor-based detection algorithms include: LOF (local anomaly factor detection algorithm), COF (outlier factor detection algorithm), CBLOF (local anomaly factor detection algorithm), LOCI (fast outlier detection algorithm based on local correlation integral), etc.; probability-based detection algorithms include fastABOD (fast angle-based outlier detection algorithm), SOS (random outlier detection algorithm); integration-based detection algorithms include iForest (isolation forest detection algorithm), LSCP (local selection combination algorithm of parallel anomaly sets), XGBOD (extreme boosting-based detection algorithm); neural network-based detection algorithms include autoEncoder (autoencoder model), SO_GAAL (single target generative adversarial active learning model), etc.

[0037] For an embodiment of the present invention, after obtaining the attribute features corresponding to the data set to be detected, in order to detect abnormal data in the data set to be detected, it is necessary to input the attribute features into a preset abnormal detection algorithm prediction model for algorithm prediction, obtain a target abnormal detection algorithm applicable to the data set to be detected, and use the target abnormal detection algorithm to detect the abnormal data in the data set to be detected, and finally obtain the abnormal data detection result in the data set to be detected. For example, the attribute features corresponding to the income data set of employees of a certain enterprise are obtained, and the attribute features are input into a preset abnormal detection algorithm prediction model for algorithm prediction, and the target abnormal detection algorithm corresponding to the income data set is obtained as an isolation forest algorithm. Finally, the isolation forest algorithm is used to detect the abnormal data in the income data set to obtain the abnormal data detection result, thereby improving the efficiency of abnormal data detection. At the same time, this method can avoid the situation where an inappropriate abnormal detection algorithm is selected to detect the data set to be detected due to subjective factors of the staff, thereby improving the detection accuracy of abnormal data.

[0038] 103. Detect abnormal data in the to-be-detected data set using the target anomaly detection algorithm to obtain an abnormal data detection result corresponding to the to-be-detected data set.

[0039] Among them, abnormal data refers to data that has a large behavioral difference from the expected data. For example, when analyzing the monthly income of an employee, if the income of a certain month suddenly increases due to the year-end bonus, the income data of that month is abnormal data.

[0040] For the embodiment of the present invention, after determining the target anomaly detection algorithm applicable to the data set to be detected, the target anomaly detection algorithm is used to detect the data set to be detected. If the target anomaly detection algorithm is a random anomaly detection algorithm, the random anomaly detection algorithm is used to detect the data set to be detected. If the target anomaly detection algorithm is a local anomaly factor detection algorithm, the local anomaly factor detection algorithm is used to detect the data set to be detected, and finally the abnormal data detection result corresponding to the data set to be detected is obtained. For example, the data set to be detected is a monthly salary data set of an employee in 2020, where the salary income from January to December is January - 5,000 yuan, February - 5,250 yuan, March - 5 yuan,..., December - 50 50 yuan, etc., and the target anomaly detection algorithm corresponding to the wage income data set is predicted to be the isolation forest algorithm. The isolation forest algorithm is used to perform anomaly detection on the wage income data set, and the abnormal data in the data set to be detected is the wage income of 5 yuan in March. Therefore, the preset anomaly detection algorithm prediction model is used to predict the target anomaly detection algorithm applicable to the data set to be detected, and the predicted target anomaly detection algorithm is used to detect the abnormal data in the data set to be detected, so as to obtain the anomaly detection result corresponding to the data set to be detected, thereby improving the efficiency of abnormal data detection. At the same time, this method can avoid the situation where an inappropriate anomaly detection algorithm is selected to detect the data set to be detected due to subjective factors of the staff, thereby improving the detection accuracy of abnormal data.

[0041] According to a method for detecting abnormal data provided by the present invention, compared with the current method of detecting abnormal data in a data set by manually selecting anomaly detection algorithms, the present invention obtains attribute features corresponding to the data set to be detected; and inputs the attribute features into a preset anomaly detection algorithm prediction model for algorithm prediction to obtain a target anomaly detection algorithm applicable to the data set to be detected; finally, the target anomaly detection algorithm is used to detect the abnormal data in the data set to be detected to obtain an abnormal data detection result corresponding to the data set to be detected, thereby using the preset anomaly detection algorithm prediction model to predict the abnormal detection algorithm applicable to the data set to be detected, and using the predicted anomaly detection algorithm to detect the abnormal data in the data set to be detected to obtain an abnormal detection result corresponding to the data set to be detected, thereby improving the efficiency of abnormal data detection. At the same time, this method can avoid the situation where an inappropriate anomaly detection algorithm is selected to detect the data set to be detected due to subjective factors of the staff, thereby improving the detection accuracy of abnormal data.

[0042] Further, in order to better illustrate the above process of detecting abnormal data, as a refinement and extension of the above embodiment, the embodiment of the present invention provides another method for detecting abnormal data, such as Figure 2 As shown, the method includes:

[0043] 201. Obtain attribute features corresponding to the data set to be detected.

[0044] For the embodiment of the present invention, in order to detect abnormal data in the data set to be monitored, it is first necessary to supplement the missing data in the data set to be detected. Based on this, the method includes: determining the known data in the data set to be detected; using a preset interpolation algorithm to interpolate the known data to obtain each data in the data set to be detected.

[0045] Among them, the preset difference algorithm can be specifically a preset Kriging interpolation algorithm. Specifically, first determine the position information of the known data and the unknown data in space, then calculate the distance between the known data position and the unknown data position, and determine the weight coefficient corresponding to the known data according to the distance. According to the weight coefficient, perform weighted summation on the known data to obtain the unknown data. The known data and the unknown data constitute each data in the data set to be detected. The specific calculation formula is as follows:

[0046]

[0047] in, is unknown data, λ i is the weight coefficient corresponding to the known data, z i is known data.

[0048] Furthermore, after interpolation processing is performed on known data in the data set to be detected, it is necessary to obtain attribute features corresponding to the data set to be detected after the interpolation processing. Based on this, step 201 specifically includes: expanding the data in the data set to be detected to obtain an expanded data set corresponding to the data set to be detected; extracting attribute features of the expanded data set to obtain an attribute feature data set corresponding to the expanded data set; performing dimensionality reduction processing on the attribute feature data set using a preset dimensionality reduction algorithm to obtain attribute features corresponding to the data set to be detected.

[0049] Specifically, firstly, the data set to be detected is subjected to automated feature engineering processing, wherein the automated feature engineering includes three dimensions: basic feature engineering, time series feature processing, and behavior sequence feature processing. The data set to be detected is subjected to feature processing in the above three dimensions to obtain an extended data set corresponding to the data set to be detected, wherein the amount of data in the extended data set is greater than the amount of data in the data set to be detected. Then, the extended data set is subjected to attribute feature engineering processing, i.e., the attribute features corresponding to each data in the extended data set are extracted to obtain an attribute feature data set corresponding to the extended data set. The attribute features include attributes such as the mean, maximum value, minimum value, coefficient of fluctuation, and standard deviation value corresponding to each data. At the same time, in order to ensure the availability of the attribute features in the attribute feature data set, it is necessary to perform dimensionality reduction on the attribute data in the attribute feature data set. Processing is performed to finally obtain the attribute features corresponding to the data set to be detected, wherein the method for dimensionality reduction processing of the attribute feature data set includes: using each feature data recorded in the attribute feature data set to form a first feature matrix; performing zero mean processing on each row of feature data in the first feature matrix to obtain a processed first feature matrix; determining the covariance matrix corresponding to the processed first feature matrix, and calculating each eigenvalue in the covariance matrix and its corresponding eigenvector; sorting the eigenvectors by row from large to small according to their corresponding eigenvalues, and extracting the eigenvectors whose sorting results are within a preset range, and forming them into a second feature matrix; multiplying the first feature matrix by the second feature matrix to obtain the attribute feature matrix corresponding to the data set to be detected, wherein the attribute feature matrix records the attribute features corresponding to the data set to be detected.

[0050] Specifically, each feature data recorded in the attribute feature data set is composed into a first feature matrix, and the mean corresponding to each row of feature data in the first feature matrix is ​​calculated, and each feature data in each row of feature data is respectively subtracted from the mean, that is, each row of feature data in the first feature matrix is ​​subjected to zero mean processing to obtain a processed first feature matrix, and then a covariance matrix corresponding to the processed first feature matrix is ​​determined, and each eigenvalue and its corresponding eigenvector in the covariance matrix are calculated, and the eigenvectors are sorted from top to bottom according to the size of their corresponding eigenvalues, and the eigenvectors of the first n rows are extracted, and the eigenvectors of the first n rows are composed of a second feature matrix, and finally the first feature matrix is ​​multiplied by the second feature matrix to obtain an attribute feature matrix corresponding to the data set to be detected, the attribute feature matrix records the attribute features corresponding to the data set to be detected, and the attribute features are input into a preset anomaly detection algorithm prediction model for algorithm prediction to obtain a target anomaly detection algorithm applicable to the data set to be detected, and finally the target anomaly detection is used to detect the abnormal data in the data set to be detected to obtain the abnormal data detection result corresponding to the data set to be detected.

[0051] 202. Input the attribute features into a multilayer perceptron, and extract features output by the last fully connected layer in the multilayer perceptron.

[0052] For the embodiment of the present invention, before the attribute feature number is input into the multilayer perceptron, in order to improve the prediction accuracy of the multilayer perceptron, it is necessary to train the multilayer perceptron. Based on this, the method includes: obtaining the attribute features corresponding to the sample data set; using multiple anomaly detection algorithms to perform abnormal data detection on the sample data set respectively, and obtaining multiple abnormal data detection results corresponding to the sample data set; according to the multiple abnormal data detection results corresponding to the sample data set, determining the detection accuracy corresponding to the multiple anomaly detection algorithms respectively; based on the detection accuracy, determining the target sample anomaly detection algorithm applicable to the sample data set; using the attribute features corresponding to the sample data set and the target sample anomaly detection algorithm as training sets, and using a neural network algorithm to train the multilayer perceptron.

[0053] Specifically, firstly, a sample data set is obtained, and the sample data set can be collected online, or a desensitized data set in an enterprise can be obtained, wherein the sample data set contains original abnormal data, each original abnormal data is labeled, and each sample data set has a corresponding name, and then the sample data set is processed by automated feature engineering to obtain an expanded data set, and at the same time, the expanded data set is processed by attribute feature engineering to obtain an attribute feature data set, and then the data in the attribute feature data set is subjected to dimensionality reduction processing to obtain the attribute features corresponding to the sample data set, and the sample data set is respectively subjected to abnormal data detection using multiple anomaly detection algorithms to obtain multiple anomaly detection results corresponding to the sample data set, and based on the original abnormal data with labels and the abnormal data detected by the multiple anomaly detection algorithms, the detection accuracies corresponding to the multiple anomaly detection algorithms are determined, and the maximum detection accuracy is determined among the multiple detection accuracies, and the anomaly detection algorithm corresponding to the maximum detection accuracy is determined as the target sample anomaly detection algorithm, for example, the original abnormal data corresponding to the data set to be detected is the first data, the third data, and the sixth data, and the data set to be detected is detected using the first anomaly detection algorithm, and the abnormal data obtained is the third data. The first and seventh data are obtained, and the corresponding detection accuracy is 33%. The second anomaly detection algorithm is used to detect the data set to be detected, and the abnormal data obtained are the first and sixth data, and the corresponding detection accuracy is 67%. It can be seen that the second anomaly detection algorithm has the highest accuracy, so the second anomaly detection algorithm is determined as the target sample anomaly detection algorithm corresponding to the sample data set. In summary, the target sample anomaly detection algorithms corresponding to multiple sample data sets can be determined, and the attribute features and target sample anomaly detection algorithms corresponding to multiple sample data sets are sorted into a table form. Each row in the table represents a sample data set, and each row contains the attribute features corresponding to the sample data set. The last column of each row is the target sample anomaly detection algorithm applicable to the sample data set. The sample data set is split into a training sample data set and a test sample data set. The attribute features and target sample anomaly detection algorithm corresponding to the training sample data set are input into a multilayer perceptron to continuously train it, and then the attribute features and target sample anomaly detection algorithm corresponding to the test sample data set are input into the trained multilayer perceptron to continuously test it, and the parameters of the multilayer perceptron are adjusted according to the test results to obtain the optimal multilayer perceptron.

[0054] Furthermore, after the multilayer perceptron is trained, it is necessary to predict the anomaly detection algorithm applicable to the data set to be detected based on the multilayer perceptron. The specific prediction method is to input the attribute features corresponding to the data set to be detected into the hidden layer through the input layer of the multilayer perceptron, and the result output by the hidden layer is:

[0055] f(W1x+b1)

[0056] Among them, the output result is the attribute feature after the attribute feature is fully connected through the multi-layer perceptron and before entering the softmax layer. For different anomaly detection algorithms, x is the attribute feature, w1 is the weight of the hidden layer, which is also the connection coefficient of the multi-layer perceptron, b1 is the bias coefficient of the hidden layer, and the f function can usually use the sigmoid function or the tanh function, as shown below:

[0057] sigmoid(x)=1 / (1+e -x )

[0058] tanh(x)=(e x -e -x ) / (e 1 +e -x )

[0059] Furthermore, after the attribute features are input into the hidden layer through the input layer of the multi-layer perceptron model and the output result of the hidden layer is obtained, the result is input into the softmax layer to obtain the target anomaly detection algorithm applicable to the data set to be detected.

[0060] 203. Input the features output by the last fully connected layer into the softmax layer in the multi-layer perceptron to obtain a target anomaly detection algorithm applicable to the data set to be detected.

[0061] For the embodiment of the present invention, after extracting the features output by the last fully connected layer in the multilayer perceptron, the result is input into the output layer, i.e., the softmax layer of the multilayer perceptron, and the target anomaly detection algorithm is recommended through the output layer. The recommendation result obtained is:

[0062] softmax(W2f(W1x+b1)+b2)

[0063] Among them, W2 is the weight coefficient of the output layer, b2 is the bias coefficient of the output layer, and the output layer of the multilayer perceptron model can output the target anomaly detection algorithm applicable to the data set to be detected. The recommended algorithm is essentially the classification probability of the anomaly detection algorithm corresponding to the data set to be detected. The anomaly detection algorithm corresponding to the maximum classification probability is determined as the target anomaly detection algorithm corresponding to the data set to be detected. Finally, the abnormal data in the data set to be detected is detected based on the target anomaly detection algorithm to obtain the abnormal data detection result corresponding to the data set to be detected.

[0064] 204. Detect abnormal data in the to-be-detected data set using the target anomaly detection algorithm to obtain an abnormal data detection result corresponding to the to-be-detected data set.

[0065] Among them, the target anomaly detection algorithm includes a clustering algorithm. For the embodiment of the present invention, if the target anomaly detection algorithm applicable to the data set to be detected is a clustering algorithm, it is necessary to use the clustering algorithm to detect the abnormal data in the data set to be detected. Based on this, step 204 specifically includes: determining the feature vector corresponding to each data in the data set to be detected; initializing the centroid vectors corresponding to different clusters; calculating the distance between the feature vector and the centroid vectors corresponding to the different clusters, and dividing the each data into the different clusters based on the distances corresponding to the different clusters; based on the feature vectors corresponding to each data in the different clusters, obtaining the updated centroid vectors corresponding to the different clusters; based on the updated centroid vectors, re-dividing the each data into the different clusters until the updated centroid vector does not change; determining the target data that is not in any cluster, and determining the target data as the abnormal data in the data set to be detected.

[0066] Specifically, the feature vector corresponding to each data in the data set to be detected is determined, and at the same time, the centroid vectors corresponding to the initial centroids corresponding to the K clusters are selected. For the feature vectors corresponding to each data, the distances from each feature vector to the K centroid vectors are calculated, and each feature vector is assigned to the cluster corresponding to the nearest centroid vector. Then, for each cluster, the centroid of each cluster and its corresponding centroid vector are recalculated, and the data are re-divided into different clusters, so as to continuously divide the data until the position of the centroid does not change, that is, the centroid vector does not change, and finally the target data that is not in any cluster is determined, and the target data is the abnormal data in the data to be detected.

[0067] According to another abnormal data detection method provided by the present invention, compared with the current method of detecting abnormal data in a data set by manually selecting anomaly detection algorithms, the present invention obtains attribute features corresponding to the data set to be detected; and inputs the attribute features into a preset anomaly detection algorithm prediction model for algorithm prediction to obtain a target anomaly detection algorithm applicable to the data set to be detected; finally, the target anomaly detection algorithm is used to detect the abnormal data in the data set to be detected to obtain an abnormal data detection result corresponding to the data set to be detected, thereby using the preset anomaly detection algorithm prediction model to predict the abnormal detection algorithm applicable to the data set to be detected, and using the predicted anomaly detection algorithm to detect the abnormal data in the data set to be detected to obtain an abnormal detection result corresponding to the data set to be detected, thereby improving the efficiency of abnormal data detection. At the same time, this method can avoid the situation where an inappropriate anomaly detection algorithm is selected to detect the data set to be detected due to subjective factors of the staff, thereby improving the detection accuracy of abnormal data.

[0068] Further, as Figure 1 The specific implementation of the present invention provides a device for detecting abnormal data, such as Figure 3 As shown, the device includes: an acquisition unit 31, an algorithm prediction unit 32 and an abnormality detection unit 33.

[0069] The acquisition unit 31 may be used to acquire attribute features corresponding to the data set to be detected.

[0070] The algorithm prediction unit 32 may be used to input the attribute features into a preset anomaly detection algorithm prediction model to perform algorithm prediction, and obtain a target anomaly detection algorithm applicable to the data set to be detected.

[0071] The anomaly detection unit 33 may be configured to detect the anomaly data in the to-be-detected data set using the target anomaly detection algorithm to obtain an anomaly data detection result corresponding to the to-be-detected data set.

[0072] In specific application scenarios, in order to obtain the attribute features corresponding to the data set to be detected, such as Figure 4 As shown, the acquisition unit 31 includes an expansion module 311 , an extraction module 312 and a dimensionality reduction module 313 .

[0073] The expansion module 311 may be used to expand the data in the to-be-detected data set to obtain an expanded data set corresponding to the to-be-detected data set.

[0074] The extraction module 312 may be used to extract attribute features from the expanded data set to obtain an attribute feature data set corresponding to the expanded data set.

[0075] The dimension reduction module 313 may be used to perform dimension reduction processing on the attribute feature data set using a preset dimension reduction algorithm to obtain attribute features corresponding to the data set to be detected.

[0076] In a specific application scenario, in order to use a preset dimensionality reduction algorithm to perform dimensionality reduction processing on the attribute feature data set to obtain the attribute features corresponding to the data set to be detected, the dimensionality reduction module 313 includes a composition submodule, a processing submodule, a calculation submodule, an extraction submodule and a multiplication submodule.

[0077] The composition submodule may be used to compose a first feature matrix using each feature data recorded in the attribute feature data set.

[0078] The processing submodule may be used to perform zero mean processing on each row of feature data in the first feature matrix to obtain a processed first feature matrix.

[0079] The calculation submodule may be used to determine the covariance matrix corresponding to the processed first feature matrix, and calculate each eigenvalue in the covariance matrix and its corresponding eigenvector.

[0080] The extraction submodule may be used to sort the feature vectors by row from large to small according to their corresponding feature values, and extract feature vectors whose sorting results are within a preset range to form a second feature matrix.

[0081] The multiplication submodule may be used to multiply the first feature matrix and the second feature matrix to obtain an attribute feature matrix corresponding to the data set to be detected, wherein the attribute feature matrix records the attribute features corresponding to the data set to be detected.

[0082] In a specific application scenario, in order to train the preset anomaly detection algorithm prediction model, the device further includes: a determination unit 34 and a training unit 35.

[0083] The acquisition unit 31 may also be used to acquire attribute features corresponding to the sample data set.

[0084] The anomaly detection unit 33 may also be configured to perform anomaly detection on the sample data set using multiple anomaly detection algorithms to obtain multiple anomaly detection results corresponding to the sample data set.

[0085] The determination unit 34 may be configured to determine the detection accuracies respectively corresponding to the multiple anomaly detection algorithms according to the multiple anomaly data detection results corresponding to the sample data set.

[0086] The determining unit 34 may be specifically configured to determine a target sample anomaly detection algorithm applicable to the sample data set based on the detection accuracy.

[0087] The training unit 35 may be used to use the attribute features corresponding to the sample data set and the target sample anomaly detection algorithm as a training set, and train the preset anomaly detection algorithm prediction model using a neural network algorithm.

[0088] In a specific application scenario, in order to determine the target anomaly detection algorithm applicable to the data set to be detected, the algorithm prediction unit 32 can be specifically used to input the attribute features into the multilayer perceptron, extract the features of the output of the last fully connected layer in the multilayer perceptron; input the features of the output of the last fully connected layer into the softmax layer in the multilayer perceptron, and obtain the target anomaly detection algorithm applicable to the data set to be detected.

[0089] In a specific application scenario, in order to use the target anomaly detection algorithm to detect abnormal data in the data set to be detected, the anomaly detection unit 33 includes a determination module 331, an initialization module 332, a division module 333, an acquisition module 334 and an update module 335.

[0090] The determination module 331 may be used to determine a feature vector corresponding to each data in the to-be-detected data set.

[0091] The initialization module 332 may be used to initialize centroid vectors corresponding to different clusters.

[0092] The division module 333 may be used to calculate the distance between the feature vector and the centroid vectors corresponding to the different clusters, and divide the data into the different clusters based on the distances corresponding to the different clusters.

[0093] The acquisition module 334 may be used to obtain updated centroid vectors corresponding to the different clusters based on the feature vectors corresponding to the data in the different clusters.

[0094] The updating module 335 may be used to re-divide the various data into the different clusters based on the updated centroid vector until the updated centroid vector does not change.

[0095] The determination module 331 may be specifically used to determine target data that is not in any cluster, and determine the target data as abnormal data in the data set to be detected.

[0096] In a specific application scenario, in order to supplement the missing data in the data set to be detected, the device further includes an interpolation unit 36.

[0097] The determining unit 34 may also be used to determine known data in the data set to be detected.

[0098] The interpolation unit 36 ​​may be configured to perform interpolation processing on the known data using a preset interpolation algorithm to obtain each data in the data set to be detected.

[0099] It should be noted that for other corresponding descriptions of the functional modules involved in the abnormal data detection device provided in the embodiment of the present invention, reference can be made to Figure 1 The corresponding description of the method shown will not be repeated here.

[0100] Based on the above Figure 1 The method shown, accordingly, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented: obtaining attribute features corresponding to the data set to be detected; inputting the attribute features into a preset anomaly detection algorithm prediction model for algorithm prediction, and obtaining a target anomaly detection algorithm applicable to the data set to be detected; using the target anomaly detection algorithm to detect abnormal data in the data set to be detected, and obtaining an abnormal data detection result corresponding to the data set to be detected.

[0101] Based on the above Figure 1 The method shown and Figure 3 The embodiment of the device shown in the figure, the embodiment of the present invention also provides a physical structure diagram of a computer device, such as Figure 5 As shown, the computer device includes: a processor 41, a memory 42, and a computer program stored in the memory 42 and executable on the processor, wherein the memory 42 and the processor 41 are both arranged on a bus 43, and the processor 41 implements the following steps when executing the program: obtaining attribute features corresponding to the data set to be detected; inputting the attribute features into a preset anomaly detection algorithm prediction model for algorithm prediction to obtain a target anomaly detection algorithm applicable to the data set to be detected; and using the target anomaly detection algorithm to detect abnormal data in the data set to be detected to obtain an abnormal data detection result corresponding to the data set to be detected.

[0102] Through the technical solution of the present invention, the present invention obtains the attribute features corresponding to the data set to be detected; and inputs the attribute features into a preset anomaly detection algorithm prediction model for algorithm prediction to obtain a target anomaly detection algorithm applicable to the data set to be detected; finally, the target anomaly detection algorithm is used to detect the abnormal data in the data set to be detected to obtain the abnormal data detection result corresponding to the data set to be detected, thereby using the preset anomaly detection algorithm prediction model to predict the anomaly detection algorithm applicable to the data set to be detected, and using the predicted anomaly detection algorithm to detect the abnormal data in the data set to be detected to obtain the anomaly detection result corresponding to the data set to be detected, thereby improving the efficiency of abnormal data detection. At the same time, this method can avoid the situation where an inappropriate anomaly detection algorithm is selected to detect the data set to be detected due to subjective factors of the staff, thereby improving the detection accuracy of abnormal data.

[0103] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0104] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting abnormal data, characterized in that: include: Acquire attribute features corresponding to the data set to be detected, wherein the data set to be detected includes: an income data set of enterprise employees; Input the attribute features of the income data set of the enterprise employees into the multilayer perceptron in the preset anomaly detection algorithm prediction model, and extract the features of the output of the last fully connected layer in the multilayer perceptron; input the features of the output of the last fully connected layer into the softmax layer in the multilayer perceptron to obtain the target anomaly detection algorithm applicable to the data set to be detected, wherein the preset anomaly detection algorithm prediction model is pre-trained based on the training set, and the specific training method includes: obtaining the attribute features corresponding to the income sample data set of the enterprise employees; using multiple anomaly detection algorithms to perform abnormal data detection on the sample data set respectively, and obtaining multiple abnormal data detection results corresponding to the sample data set; according to the multiple abnormal data detection results corresponding to the sample data set, determining the detection accuracy corresponding to the multiple anomaly detection algorithms respectively; based on the detection accuracy, determining the target sample anomaly detection algorithm applicable to the sample data set; using the attribute features corresponding to the sample data set and the target sample anomaly detection algorithm as the training set, and using the neural network algorithm to train the preset anomaly detection algorithm prediction model; The target anomaly detection algorithm is used to detect the abnormal data in the data set to be detected, and the abnormal data detection result corresponding to the data set to be detected is obtained, wherein, if the target anomaly detection algorithm is a clustering algorithm, the method for detecting abnormal data includes: determining the feature vector corresponding to each data in the data set to be detected; initializing the centroid vectors corresponding to different clusters; calculating the distance between the feature vector and the centroid vectors corresponding to the different clusters, and dividing the each data into the different clusters based on the distances corresponding to the different clusters; based on the feature vectors corresponding to each data in the different clusters, obtaining the updated centroid vectors corresponding to the different clusters; based on the updated centroid vectors, re-dividing the each data into the different clusters until the updated centroid vector does not change; determining the target data that is not in any cluster, and determining the target data as the abnormal data in the data set to be detected.

2. The method according to claim 1, characterized in that The step of obtaining the attribute features corresponding to the data set to be detected includes: Expanding the data in the to-be-detected data set to obtain an expanded data set corresponding to the to-be-detected data set; Extracting attribute features from the expanded data set to obtain an attribute feature data set corresponding to the expanded data set; The attribute feature data set is subjected to dimensionality reduction processing using a preset dimensionality reduction algorithm to obtain the attribute features corresponding to the data set to be detected.

3. The method according to claim 2, characterized in that The method of performing dimensionality reduction processing on the attribute feature data set by using a preset dimensionality reduction algorithm to obtain the attribute features corresponding to the data set to be detected includes: Using each feature data recorded in the attribute feature data set to form a first feature matrix; Performing zero mean processing on each row of feature data in the first feature matrix to obtain a processed first feature matrix; Determine a covariance matrix corresponding to the processed first feature matrix, and calculate each eigenvalue in the covariance matrix and its corresponding eigenvector; Sorting the eigenvectors in rows from large to small according to their corresponding eigenvalues, extracting eigenvectors whose sorting results are within a preset range, and forming them into a second eigenmatrix; The first feature matrix is ​​multiplied by the second feature matrix to obtain an attribute feature matrix corresponding to the data set to be detected, wherein the attribute feature matrix records the attribute features corresponding to the data set to be detected.

4. The method according to claim 1, characterized in that Before obtaining the attribute features corresponding to the data set to be detected, the method further includes: Determining known data in the data set to be detected; The known data are interpolated using a preset interpolation algorithm to obtain each data in the data set to be detected.

5. A device for detecting abnormal data, characterized in that: include: An acquisition unit is used to acquire attribute features corresponding to the data set to be detected, wherein the data set to be detected includes: an income data set of enterprise employees; An algorithm prediction unit is used to input the attribute characteristics of the income data set of the enterprise employees into the multilayer perceptron in the preset anomaly detection algorithm prediction model, and extract the characteristics of the output of the last fully connected layer in the multilayer perceptron; input the characteristics of the output of the last fully connected layer into the softmax layer in the multilayer perceptron to obtain a target anomaly detection algorithm applicable to the data set to be detected, wherein the preset anomaly detection algorithm prediction model is pre-trained based on a training set, and the specific training method includes: obtaining the attribute characteristics corresponding to the income sample data set of the enterprise employees; using multiple anomaly detection algorithms to perform abnormal data detection on the sample data set respectively, and obtaining multiple abnormal data detection results corresponding to the sample data set; according to the multiple abnormal data detection results corresponding to the sample data set, determining the detection accuracy corresponding to the multiple anomaly detection algorithms respectively; based on the detection accuracy, determining the target sample anomaly detection algorithm applicable to the sample data set; using the attribute characteristics corresponding to the sample data set and the target sample anomaly detection algorithm as a training set, and using a neural network algorithm to train the preset anomaly detection algorithm prediction model; An anomaly detection unit is used to detect the abnormal data in the data set to be detected by using the target anomaly detection algorithm to obtain the abnormal data detection result corresponding to the data set to be detected, wherein if the target anomaly detection algorithm is a clustering algorithm, the method for detecting abnormal data includes: determining the feature vector corresponding to each data in the data set to be detected; initializing the centroid vectors corresponding to different clusters; calculating the distance between the feature vector and the centroid vectors corresponding to the different clusters, and dividing the each data into the different clusters based on the distances corresponding to the different clusters; based on the feature vectors corresponding to each data in the different clusters, obtaining the updated centroid vectors corresponding to the different clusters; based on the updated centroid vectors, re-dividing the each data into the different clusters until the updated centroid vector does not change; determining the target data that is not in any cluster, and determining the target data as the abnormal data in the data set to be detected.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images