A method and system for detecting multi-dimensional anomalies in data

Abstract

The present invention relates to a method and system for detecting multi-dimensional anomalies in data. The method includes: Step S1: Obtain the original data, and determine whether the original data is tabular data. If it is not tabular data, convert the original data into tabular data; Step S2: Divide the tabular data into a training set and a test set and perform preprocessing on them respectively; Step S3: Construct an autoencoder, and improve the loss function of the autoencoder to achieve the conversion of the autoencoder from completely reconstructing the input data to moderately reconstructing it; Step S4: Based on the improved loss function, train the autoencoder with the training set to obtain a trained autoencoder; Step S5: Input the test set into the trained autoencoder to detect the anomaly points in the multi-dimensional tabular data. By improving the loss function of the autoencoder, the improved autoencoder in the present invention has a better detection effect on multi-dimensional anomalies in data.

Description

Technical Field

[0001] The present invention relates to the technical field of data anomaly detection, and in particular to a method and system for detecting multi-dimensional data anomalies. Background Art

[0002] Anomaly detection refers to identifying data points that deviate significantly from normal data. So far, anomaly detection has been widely applied in various fields of science and technology, such as credit card fraud detection, intrusion detection, and text anomaly detection. The common anomaly detection schemes are as follows:

[0003] (1) Traditional statistical anomaly detection schemes: Common statistical anomaly detection schemes such as the "3σ principle" and box plots usually perform anomaly detection based on the values of data in a single dimension. When the value of data in a single dimension is large or small, the data is identified as an anomaly. When the data dimension is high, the efficiency of this scheme is low. In addition, this scheme ignores the internal relationship between different dimensions of data during the anomaly detection process.

[0004] (2) K-Nearest Neighbor anomaly detection algorithm (KNN): The idea of the KNN algorithm is that anomaly points must be far from most of the sample points. Based on this idea, it is necessary to calculate the average distance between each sample point and its nearest K samples in turn, and then compare the calculated distance with a threshold. If it is greater than the threshold, it is considered an anomaly point. This scheme requires a large amount of calculation of the distance between different sample points, so the detection efficiency for high-dimensional data is low.

[0005] (3) Local Outlier Factor anomaly detection algorithm (LOF): The basic idea of the LOF algorithm is to calculate a local reachability density for each data point according to the data density around the data point, and then further calculate an outlier factor for each data point through the local reachability density. This outlier factor indicates the degree of outlier of a data point. The larger the factor value, the higher the degree of outlier; the smaller the factor value, the lower the degree of outlier. Finally, the top n points with the highest degree of outlier are output as anomaly points. This scheme needs to calculate the local reachability density of each data point, and when the data volume is large or the data dimension is high, the timeliness of this scheme is low.

[0006] (4) Autoencoder method: As a neural network model, autoencoder has been widely used in anomaly detection due to its obvious advantages in processing high-dimensional and nonlinear data sets. The autoencoder first compresses the input data into a low-dimensional space, and then reconstructs the compressed data into the original space. The difference between each data point after reconstruction by the autoencoder and its corresponding original data is called the reconstruction error. In the process of autoencoder anomaly detection, the autoencoder is first trained with the data of the training set. By minimizing the loss function, the autoencoder can be driven to learn the main intrinsic information of the data set. Then, the data of the test set is reconstructed with the trained autoencoder, and the degree of anomaly of each data point can be judged based on the reconstruction error. The autoencoder has a natural ability to identify the strong influencing points (usually points that destroy the intrinsic relationship between certain dimensions) of the data set, but the detection results of high leverage points (usually points far from the mean) are full of uncertainty.

[0007] When performing anomaly detection on multi-dimensional tabular data, traditional statistical anomaly detection schemes have good detection effects on data points that are abnormal in a single dimension. However, there is uncertainty in the detection results for data points that are normal in a single dimension but are abnormal in multiple dimensions. The KNN and LOF algorithms judge the degree of abnormality of data points based on distance and density, respectively. When the dimension of the data set is high, the timeliness of these two algorithms is low. In addition, the KNN and LOF algorithms also have uncertainty in the detection results of strongly influential points. The autoencoder algorithm has obvious advantages in processing high-dimensional and nonlinear data sets, but existing autoencoder algorithms usually use loss functions such as MSE and MAE, which have a lot of uncertainty in the detection results of high-leverage points with multi-dimensional joint anomalies:

[0008] (1) For each input data point, if its value in a certain dimension exceeds the normal range, it will be identified as an outlier. However, this identification method is closer to independent detection of each dimension rather than multi-dimensional joint detection of anomalies. Figure 2 As shown in the figure, for a binary normally distributed dataset, a conventional autoencoder algorithm is used for anomaly detection. When the hidden layer dimension is set to 2 and the anomaly ratio is set to 5%, the data points outside the white circle (the white circle is added for the convenience of observation) are identified as anomalies. It can be seen that all the high-leverage points marked as anomalies are due to the fact that the value of a single dimension exceeds the normal range, forming the outer edge of the square. In fact, the anomalies of the joint distribution should be on the outer edge of the circle or ellipse, not as shown in the figure. Figure 2 Therefore, existing autoencoder methods lack the ability to detect multi-dimensional joint anomalies.

[0009] (2) If the reconstruction effects of each principal component direction of the dataset are different, more high-leverage points will be detected in the principal component direction with a poorer reconstruction effect, while high-leverage points in the principal component direction with a better reconstruction effect may not even be detected. As Figure 3 shown, the conventional autoencoder algorithm is used to detect anomalies in a dataset with a bivariate normal distribution. The directions of the two perpendicular lines are respectively the two principal component directions of the dataset. Figure 4 and Figure 5 are respectively the reconstruction error distributions of the dataset in the two principal component directions. It can be seen that the reconstruction result in the "Parameter2" direction is better than that in the "Parameter1" direction. When the hidden layer dimension is set to 2 and the anomaly ratio is set to 5%, it can be found from Figure 3 that most of the detected high-leverage points are located in the "Parameter1" direction. Therefore, the existing autoencoder method has a dimensional preference in anomaly detection and will ignore anomalies in other dimensions.

[0010] Figure 6 and Figure 7 are two other datasets with a normal distribution. Figure 6 The correlation coefficient between the two dimensions of the dataset is relatively large. Figure 7 is a 3D dataset with an intrinsic dimension of 2, and each dimension of the dataset follows a normal distribution. The conventional autoencoder anomaly detection scheme is used to detect anomalies in the two datasets. When the hidden layer dimension is set to 2 and the anomaly ratio is set to 5%, it can be seen that the above two problems also exist in the detection results. Summary of the Invention

[0011] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the anomaly detection algorithm in the prior art has poor detection effect on multi-dimensional data.

[0012] To solve the above technical problem, the present invention provides a method for detecting multi-dimensional anomalies in data, including:

[0013] Step S1: Obtain the original data, and determine whether the original data is tabular data. If it is not tabular data, convert the original data into tabular data;

[0014] Step S2: Divide the tabular data into a training set and a test set and perform preprocessing on them respectively;

[0015] Step S3: Construct an autoencoder, and improve the loss function of the autoencoder to achieve the conversion of the autoencoder from completely reconstructing the input data to moderately reconstructing it;

[0016] Step S4: Based on the improved loss function, train the autoencoder with the preprocessed training set to obtain a trained autoencoder;

[0017] Step S5: Input the preprocessed test set into the trained autoencoder to detect outliers in the multidimensional table data.

[0018] In one embodiment of the present invention, the preprocessing in step S2 is normalization processing, and the formula is:

[0019]

[0020] Among them, x ij Represents the i-th data on the j-th dimension in the table after normalization. Represents the i-th data on the j-th dimension in the original table, m is the total dimension of the data, and n is the total number of data points.

[0021] In one embodiment of the present invention, the autoencoder in step S3 is a fully connected autoencoder, which includes an encoder and a decoder. The encoder includes an input layer, an encoding layer, and a hidden layer connected in sequence, and the decoder includes a hidden layer, a decoding layer, and an output layer connected in sequence. Each layer in the fully connected autoencoder includes a number of neurons, the neurons between adjacent layers are interconnected, and the number of neurons between the input layer and the output layer is equal; wherein, the encoder is used to compress the input data and learn the information features within the data set; the decoder is used to decode the compressed data to the original dimension of the input data.

[0022] In one embodiment of the present invention, the number of the encoding layers is several layers, the number of the decoding layers is several layers, and the number of encoding layers is equal to the number of decoding layers. The activation function used by the encoding layer and the decoding layer is a RELU function.

[0023] In one embodiment of the present invention, the loss function of the autoencoder is improved in step S3, and the formula is:

[0024]

[0025] Among them, ω represents the connection weight between each layer of the autoencoder, b represents the bias value, and x i =(x i1 ,x i2 ,...,x im ) T Represents the i-th input data, and its corresponding output data is λ i It represents the i-th eigenvalue of the input data set after sorting from large to small. is the corresponding reconstruction result; l represents the intrinsic dimension of the data set; is the reconstruction error term MSE, the loss function is the reconstruction suppression term; θ1,θ2>0 are the weights of the reconstruction error term and the reconstruction suppression term in the loss function, and β>0 is used to limit the degree of reconstruction of the dataset by the autoencoder.

[0026] In one embodiment of the present invention, the method of training the autoencoder using the preprocessed training set in step S4 includes:

[0027] The autoencoder aims to minimize the reconstruction error term MSE of the loss function to drive the autoencoder to learn the main information inside the data and reconstruct the input data, and limits the autoencoder's complete reconstruction of the input data through the reconstruction inhibition term; under the joint action of the reconstruction error term MSE of the loss function and the reconstruction inhibition term, the autoencoder updates the connection weights ω and bias values b in the network through repeated iterative training, and finally ensures that the reconstruction errors of high leverage points and strong influence points in the data set are greater than the preset values by moderately reconstructing the input data set, where high leverage points are points far away from the mean, and strong influence points are points that destroy the intrinsic relationship between dimensions.

[0028] In one embodiment of the present invention, in step S1, determining whether the original data is tabular data, and if not, converting the original data into tabular data, the method includes:

[0029] If the original data is tabular data, data cleaning is performed on it, and the required data is screened and organized into tabular data;

[0030] If the original data is time series data, feature extraction is performed on the time series data to obtain the time domain features or frequency domain features of the data, specifically: analyzing the time domain features of the data through data characteristic values, wherein the data characteristic values include the phase and amplitude of periodic data and the mean and variance of non-periodic data; or extracting the frequency domain features of the data through data processing methods, wherein the data processing methods include Fourier transform, wavelet transform and Laplace transform; finally, each extracted time domain or frequency domain feature is used as each attribute of the tabular data and organized into tabular data.

[0031] To solve the above technical problems, the present invention provides a system for detecting multidimensional anomalies in data, comprising:

[0032] Acquisition and processing module: used to acquire original data, determine whether the original data is tabular data, and if not, convert the original data into tabular data;

[0033] Preprocessing module: used to divide the table data into a training set and a test set and preprocess them separately;

[0034] Improvement module: used to construct an autoencoder and improve the loss function of the autoencoder, so as to realize the conversion of the autoencoder from completely reconstructing the input data to moderately reconstructing it;

[0035] Training module: used to train the autoencoder based on the improved loss function through the preprocessed training set to obtain a trained autoencoder;

[0036] Detection module: used to input the preprocessed test set into the trained autoencoder to detect outliers in multi-dimensional tabular data.

[0037] To solve the above technical problems, the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the above method for detecting multi-dimensional data anomalies are realized.

[0038] To solve the above technical problems, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method for detecting multi-dimensional data anomalies are realized.

[0039] The above technical solutions of the present invention have the following advantages compared with the prior art:

[0040] By improving the loss function of the autoencoder, specifically adding a reconstruction suppression term to the loss function, the present invention realizes the conversion of the autoencoder from completely reconstructing the input data to moderately reconstructing it, so that the improved autoencoder not only has a good detection effect on strong influence points, but also has a good detection effect on high leverage points;

[0041] The autoencoder of the present invention has the ability to detect multi-dimensional data anomalies. Description of the Drawings

[0042] In order to make the content of the present invention easier to be clearly understood, the following further details the present invention according to specific embodiments of the present invention and in conjunction with the drawings.

[0043] Figure 1 is the flowchart of the method of the present invention;

[0044] Figure 2 is a schematic diagram of the anomaly detection result of a binary normal distribution data set (there are noise data points in the data set) with a diagonal covariance matrix using a conventional autoencoder in the prior art;

[0045] Figure 3 is a schematic diagram of the anomaly detection result of a binary normal distribution data set with a diagonal covariance matrix using a conventional autoencoder in the prior art;

[0046] Figure 4 It is a schematic diagram of the reconstruction error distribution of a dataset in the direction of the first principal component in the prior art;

[0047] Figure 5 It is a schematic diagram of the reconstruction error distribution of a dataset in the direction of the second principal component in the prior art;

[0048] Figure 6 It is a schematic diagram of the anomaly detection result of a binary normal distribution dataset with a non - diagonal covariance matrix using a conventional auto - encoder in the prior art;

[0049] Figure 7 It is a schematic diagram of the anomaly detection result of a dataset with an actual dimension of 3D but an intrinsic dimension of 2D using a conventional auto - encoder in the prior art;

[0050] Figure 8 It is a schematic diagram of the auto - encoder structure in an embodiment of the present invention;

[0051] Figure 9 It is a schematic diagram of the anomaly detection result of a dataset of the first normal distribution using an auto - encoder with an improved loss function in an embodiment of the present invention;

[0052] Figure 10 It is a schematic diagram of the anomaly detection result of a dataset of the second normal distribution using an auto - encoder with an improved loss function in an embodiment of the present invention;

[0053] Figure 11 It is a schematic diagram of the anomaly detection result of a dataset of the third normal distribution using an auto - encoder with an improved loss function in an embodiment of the present invention;

[0054] Figure 12 It is a schematic diagram of the anomaly detection result of a dataset of the fourth normal distribution using an auto - encoder with an improved loss function in an embodiment of the present invention;

[0055] Figure 13 It is a schematic diagram of the AUC score of the anomaly detection results of different schemes based on the dry bean dataset in an embodiment of the present invention;

[0056] Figure 14 It is a schematic diagram of the AP score of the anomaly detection results of different schemes based on the dry bean dataset in an embodiment of the present invention;

[0057] Figure 15 It is a schematic diagram of the AUC score of the anomaly detection results of different schemes based on the breast cancer dataset in an embodiment of the present invention;

[0058] Figure 16 It is a schematic diagram of the AP score of the anomaly detection results of different schemes based on the breast cancer dataset in an embodiment of the present invention;

[0059] Figure 17 It is a schematic diagram of the distribution of the mouse physiological data training set in the embodiment of the present invention;

[0060] Figure 18 It is a schematic diagram of the distribution of the mouse physiological data test set in the embodiment of the present invention;

[0061] Figure 19 It is the result of using the autoencoder with an improved loss function to detect anomalies in mouse physiological data in the embodiment of the present invention;

[0062] Figure 20 It is a schematic diagram of the AUC score of the anomaly detection results of different schemes based on mouse physiological data in the embodiment of the present invention;

[0063] Figure 21 It is a schematic diagram of the AP score of the anomaly detection results of different schemes based on mouse physiological data in the embodiment of the present invention;

[0064] Figure 22 It is a schematic diagram of the filtering result of the heart rate data in the embodiment of the present invention;

[0065] Figure 23 It is a schematic diagram of forming tabular data by extracting the corresponding values of features such as peak phase and peak value for the heart rate data in the embodiment of the present invention. Detailed implementation manners

[0066] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the illustrated embodiments are not intended to limit the present invention.

[0067] Embodiment 1

[0068] Refer to Figure 1 As shown, the present invention relates to a method for detecting multi-dimensional anomalies in data, including:

[0069] Step S1: Obtain the original data, and determine whether the original data is tabular data. If it is not tabular data, convert the original data into tabular data;

[0070] Step S2: Divide the tabular data into a training set and a test set and perform preprocessing on them respectively;

[0071] Step S3: Construct an autoencoder, and improve the loss function of the autoencoder to realize the conversion of the autoencoder from completely reconstructing the input data to moderately reconstructing it;

[0072] Step S4: Based on the improved loss function, train the autoencoder with the preprocessed training set to obtain a trained autoencoder;

[0073] Step S5: Input the preprocessed test set into the trained autoencoder to detect outliers in the multidimensional table data.

[0074] Conventional autoencoder anomaly detection algorithms commonly use loss functions such as MSE and MAE during data reconstruction. The use of MSE and MAE loss functions will drive the autoencoder to reconstruct the input data set (including high leverage points) as much as possible. As a result, the autoencoder has a very good detection effect on the strong influence points (influential points, usually refers to points that destroy the intrinsic relationship between certain dimensions) in the data set, but the detection results for high leverage points (high leverage points, usually refers to points far from the mean) have a lot of uncertainty. The present invention will add a suitable penalty term to the loss function of the autoencoder to limit the autoencoder's complete reconstruction of the data set. The purpose is to enable the autoencoder to maintain the detection effect of strong influence points (points that destroy the intrinsic relationship between dimensions) while greatly improving the detection effect of high leverage points (points far from the mean).

[0075] The following is a detailed introduction to this embodiment:

[0076] (1) Obtaining original data and determining whether the original data is tabular data is as follows:

[0077] If the original data is tabular data, it will be cleaned: for example, duplicate values will be deleted, missing values will be filled, and the required data will be further filtered and organized into usable tabular data;

[0078] If the original data is time series data, various techniques can be used to extract features to obtain the data's time-domain or frequency-domain characteristics. For example, the time-domain characteristics of the data can be analyzed by analyzing the phase and amplitude of periodic data, and the mean and variance of non-periodic data. Alternatively, the frequency-domain characteristics of the data can be extracted using techniques such as Fourier transform, wavelet transform, and Laplace transform. Finally, each extracted feature is organized into a table of data as an attribute.

[0079] For multidimensional table data, the multidimensional table data is divided into a training set and a test set, and the training set and the test set are preprocessed (i.e., normalized). The normalization formula is as follows:

[0080]

[0081] Among them, x ij Represents the i-th data on the j-th dimension in the table after normalization. Represents the i-th data on the j-th dimension in the original table, m is the total dimension of the data, and n is the total number of data points.

[0082] (2) Input the training set into the autoencoder. The fully connected autoencoder is used in this embodiment. Please refer to Figure 8 , the fully connected autoencoder includes an encoder and a decoder. The encoder includes an input layer, an encoding layer, and a hidden layer connected in sequence. The decoder includes a hidden layer, a decoding layer, and an output layer connected in sequence. The number of encoding layers is several (the number of encoding layers is adjusted according to the actual dataset), the number of decoding layers is several, and the number of encoding layers is the same as the number of decoding layers. Each layer in the fully connected autoencoder includes several neurons, and the neurons between adjacent layers are connected to each other. The number of neurons between the input layer and the output layer is equal. In short, the autoencoder is mainly composed of two parts: an encoder and a decoder. The encoder compresses the input data and learns the most important information features inside the dataset, and the decoder is responsible for restoring the compressed data to the initial dimension. The error between the input data and the output data is the reconstruction error. Among them, the dimension of the hidden layer can be set to the intrinsic dimension of the dataset. The activation functions of all encoding layers and decoding layers are selected as RELU, and the activation function of the output layer is selected as Sigmoid. In this embodiment, the loss function is improved (that is, a penalty term is added), and the expression of the loss function used is:

[0083]

[0084] Among them, ω represents the connection weight between each layer of the autoencoder, b represents the bias value, x i =(x i1 ,x i2 ,...,x im ) T represents the i-th input data, and its corresponding output data is λ i represents the i-th after sorting the eigenvalues of the input dataset from large to small, is its corresponding reconstruction result; l represents the intrinsic dimension of the dataset; in the loss function is the reconstruction error term MSE, and in the loss function is the reconstruction suppression term; θ1,θ2>0 are the weights of the reconstruction error term and the reconstruction suppression term in the loss function, and β>0 is used to limit the reconstruction degree of the autoencoder for the dataset.

[0085] During the training process, this embodiment uses the Adam optimizer to optimize the loss function. The autoencoder aims to minimize the loss function MSE to drive the autoencoder to learn the main information inside the data and reconstruct the input data. In addition, the existence of the reconstruction suppression term will limit the complete reconstruction of the input data by the autoencoder. Therefore, under the combined action of the loss function MSE and the reconstruction suppression penalty term, the autoencoder updates the connection weights ω and bias values b in the network through repeated iterative training, and finally can moderately reconstruct the input data set to ensure that the reconstruction errors of the high-leverage points and strong influence points in the data set are relatively large, which is equivalent to both types of data points being detected.

[0086] (3): Input the test set into the trained autoencoder for reconstruction and obtain the reconstruction error of each data point. Consider the data points with reconstruction errors exceeding the preset threshold as anomalies.

[0087] To specifically demonstrate the technical effects of the present invention, this embodiment applies the improved autoencoder of the present invention to detect data anomalies for 4 normal distribution data sets in the first normal distribution data set, the second normal distribution data set, the third normal distribution data set, and the fourth normal distribution data set (the first is a bivariate normal distribution data set with a diagonal covariance matrix, and there are noise data points in the data set; the second is a bivariate normal distribution data set with a diagonal covariance matrix; the third is a bivariate normal distribution data set with a non-diagonal covariance matrix; the fourth is a data set with an actual dimension of 3 but an intrinsic dimension of ˈ2, and the data set follows a normal distribution in each dimension), and sets the hidden layer dimension to 2 and the anomaly ratio to 5%. The final anomaly detection results are as Figures 9 - 12 shown. The data points outside the white circles (the white circles are drawn manually for clear observation) in the figure are the detected anomaly points. It can be seen from the figure that the high-leverage points detected in this embodiment are evenly distributed in the main component directions of the data set (taking Figure 9 and Figure 10 as examples, the main component directions are the horizontal direction and the vertical direction, and taking Figure 11 as an example, the main component direction is the diagonal direction), and can also accurately detect the high-leverage points with all dimensional values within the normal range. Therefore, the improved autoencoder successfully solves the problem that conventional autoencoders are difficult to detect high-leverage points.

[0088] The application object of this embodiment is not limited to tabular data. For time series data such as sound, current, and heart rate in daily life, this embodiment can also improve the effect of anomaly detection. Because for most time series data, techniques such as Fourier transform, spectral analysis, wavelet analysis, and Laplace transform can be used to convert their features into tabular data, and then anomaly detection is performed on the converted tabular data. The following uses the method of this embodiment to detect different types of data.

[0089] (1) To demonstrate the practical application value of the solution of the present invention, this embodiment applies two machine learning benchmark datasets: the Dry Bean Dataset and the Breast Cancer Dataset to detect the accuracy of anomaly detection of the solution of the present invention. For each dataset, the results of the present invention are compared with autoencoders with the mean squared error (MSE) loss function, autoencoders with the mean absolute error (MAE) loss function, and the Deep Autoencoding Gaussian Mixture Model (DAGMM). The accuracy of anomaly detection will be evaluated by the Area Under Curve (AUC) score and the Average Precision (AP) score. For the Dry Bean Dataset and the Breast Cancer Dataset, this embodiment sets the dimensions of the hidden layer of the autoencoder to 8 and 15 respectively.

[0090] The Dry Bean Dataset has a dimension of 17 and contains a total of 13,611 sample data. Among them, all dry beans can be divided into 7 categories. In the experiment, this embodiment randomly selects 2,000 sample data from a certain category of dry beans as normal data, and then randomly selects δ×2,000 data from the remaining 6 categories of dry beans as abnormal data, where δ∈{0.05, 0.1, 0.15, 0.2, 0.25} is the anomaly ratio. Figure 13 and Figure 14 are the AUC and AP scores of the anomaly detection results of different solutions respectively. It can be seen that the accuracy of the anomaly detection result of the solution of the present invention for the Dry Bean Dataset is higher than that of all other solutions.

[0091] The Breast Cancer Dataset is 31-dimensional data and contains a total of 569 samples. Among them, all samples can be divided into 2 categories: benign and malignant. This embodiment randomly selects 300 sample data from benign samples as normal data, and then randomly selects δ×300 data from the remaining categories as abnormal data, where δ∈{0.1,  0.2,  0.3,  0.4}. Figure 15 and Figure 16It is the accuracy evaluation of the anomaly detection results of different solutions for this data set. Although the effects of the solution of the present invention and DAGMM are not much different according to the AUC score, it can be seen from the AP score that the anomaly detection accuracy rate of the solution of the present invention is the highest.

[0092] (2) This embodiment performs anomaly detection on data from the International Mouse Phenotyping Consortium (IMPC). IMPC is committed to phenotypic analysis of 20,000 mouse mutants to facilitate further prediction of disease-causing genes for various human genetic diseases. First, data on various physiological indicators of gene knockout mice were downloaded from the IMPC official website, and then data on 2 physiological indicators (Calcium and Total protein) were selected for anomaly detection. Among them, the training set (without gene knockout) contains 6,640 samples, and the test set (with gene knockout) contains 15,699 samples. Figure 17 and Figure 18 are the distribution diagrams corresponding to the training set and test set data respectively. First, the autoencoder is trained with the training set data, and then the trained autoencoder is used to perform anomaly detection on the test set data. Figure 19 The data points outside the white circles in are the anomaly points identified by the solution of the present invention when the hidden layer dimension is set to 2 and the anomaly ratio is set to 5%. Since the distribution of the data points approximates a bivariate normal distribution, the data density corresponding to the data points closer to the edge part is smaller, and the degree of anomaly is higher. And the detected anomaly data points in this embodiment are all located in the edge part, and the accuracy rate of the detection result is relatively high. In addition, this embodiment further compares the anomaly detection effects of this solution and other solutions for this data set, Figure 20 and Figure 21 are the specific comparison results. It can be seen that the accuracy rate of the anomaly detection result of this embodiment for IMPC data is the highest.

[0093] In real life, there are also many forms of tabular data, such as the physical examination data of hospital patients, wine evaluation data, etc. Therefore, the application scope of the solution of the present invention is very wide.

[0094] (3) To demonstrate the application value of the proposed solution of the present invention for time series data, this embodiment will introduce how to convert time series data into tabular data using the heart rate data of volunteers. The main idea is to first extract features from the time series data through various technical means: for example, analyzing features such as the phase and amplitude of periodic data, or using techniques such as Fourier transform, wavelet transform, and Laplace transform to extract spectral features. Finally, each extracted feature is organized into tabular data as each attribute of the tabular data. For the heart rate data of each volunteer during the day, this embodiment selects a 6th-order Butterworth analog filter as the prototype to design a digital filter, and filters it with a filter having a cut-off frequency of 0.008, as Figure 22 shown. It can be seen that there are two to three peaks in the human heart rate during the day. Therefore, for heart rate data, its peak phase and peak value are important features. This embodiment extracts the corresponding values of features such as peak phase and peak value from the heart rate data of all volunteers and organizes them into tabular data, as Figure 23 shown. Further, this solution can be applied to perform anomaly detection on the organized tabular data, and the subsequent process will not be elaborated.

[0095] Embodiment Two

[0096] This embodiment provides a method for detecting multi-dimensional anomalies in data, including:

[0097] An acquisition and processing module: used to acquire the original data, determine whether the original data is tabular data, and if it is not tabular data, convert the original data into tabular data;

[0098] A preprocessing module: used to divide the tabular data into a training set and a test set and perform preprocessing on them respectively;

[0099] An improvement module: used to construct an autoencoder and improve the loss function of the autoencoder to achieve the conversion of the autoencoder from completely reconstructing the input data to moderately reconstructing it;

[0100] A training module: used to train the autoencoder based on the improved loss function through the preprocessed training set to obtain a trained autoencoder;

[0101] A detection module: used to input the preprocessed test set into the trained autoencoder to detect anomaly points in the multi-dimensional tabular data.

[0102] Embodiment Three [[ID=2⑨]]

[0103] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method for detecting multi-dimensional anomalies in data described in Embodiment One.

[0104] Example 4

[0105] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for detecting multi-dimensional anomalies in Embodiment 1 are implemented.

[0106] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages. For example, object-oriented programming languages such as Java and interpreted scripting languages such as JavaScript.

[0107] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the specified functions in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0108] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the specified functions in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the specified functions in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0110] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present application.

[0111] Obviously, the above embodiments are only examples given for clear illustration and are not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or variations can be made based on the above description. It is not necessary and impossible to enumerate all implementation manners here. And the obvious changes or variations derived therefrom still fall within the protection scope of the present invention.

Claims

1. A method for detecting multi-dimensional anomalies in data, characterized in that: include: Step S1: obtaining original data, determining whether the original data is tabular data, and if not, converting the original data into tabular data; Step S2: Divide the tabular data into a training set and a test set and preprocess them respectively; The preprocessing in step S2 is normalization processing, and the formula is: where x ij represents the i-th data on the j-th dimension in the normalized table, represents the i-th data on the j-th dimension in the original table, m is the total number of dimensions of the data, and n is the total number of data points; Step S3: constructing an autoencoder and improving the loss function of the autoencoder to convert the autoencoder's input data from complete reconstruction to moderate reconstruction; The autoencoder in step S3 is a fully connected autoencoder, which includes an encoder and a decoder. The encoder includes an input layer, an encoding layer, and a hidden layer connected in sequence, and the decoder includes a hidden layer, a decoding layer, and an output layer connected in sequence. Each layer in the fully connected autoencoder includes a number of neurons, and the neurons between adjacent layers are interconnected. The number of neurons between the input layer and the output layer is equal. The encoder is used to compress the input data and learn the information features within the data set; the decoder is used to decode the compressed data to the original dimension of the input data. The number of the encoding layers is a plurality of layers, the number of the decoding layers is a plurality of layers, and the number of the encoding layers is equal to the number of the decoding layers. The activation function used by the encoding layer and the decoding layer is a RELU function; In step S3, the loss function of the autoencoder is improved, and the formula is: Among them, ω represents the connection weight between each layer of the autoencoder, b represents the bias value, and x i =(x i1 , x i2 ,..., x im ) T represents the i-th input data, and its corresponding output data is λ i represents the i-th one after sorting the eigenvalues of the input data set from large to small, is its corresponding reconstruction result; l represents the intrinsic dimension of the data set; in the loss function is the reconstruction error term MSE, and in the loss function is the reconstruction suppression term; θ1, θ2 > 0 are the weights of the reconstruction error term and the reconstruction suppression term in the loss function, and β > 0 is used to limit the reconstruction degree of the autoencoder for the data set; Step S4: Based on the improved loss function, the autoencoder is trained using the preprocessed training set to obtain a trained autoencoder; Step S5: Input the preprocessed test set into the trained autoencoder to detect outliers in the multidimensional table data.

2. The detection method for multi-dimensional anomalies of data according to claim 1, characterized in that: In step S4, the autoencoder is trained using the preprocessed training set, and the method includes: The autoencoder aims to minimize the reconstruction error term MSE of the loss function to drive the autoencoder to learn the main information inside the data and reconstruct the input data, and limits the autoencoder's complete reconstruction of the input data through the reconstruction inhibition term; under the joint action of the reconstruction error term MSE of the loss function and the reconstruction inhibition term, the autoencoder updates the connection weights ω and bias values b in the network through repeated iterative training, and finally ensures that the reconstruction errors of high leverage points and strong influence points in the data set are greater than the preset values by moderately reconstructing the input data set, where high leverage points are points far away from the mean, and strong influence points are points that destroy the intrinsic relationship between dimensions.

3. The detection method for multi-dimensional data anomalies according to claim 1, characterized in that: In step S1, it is determined whether the original data is tabular data. If not, the original data is converted into tabular data. The method includes: If the original data is tabular data, data cleaning is performed on it, and the required data is screened and organized into tabular data; If the original data is time - series data, perform feature extraction on the time - series data to obtain the time - domain features or frequency - domain features of the data. Specifically: Analyze the time - domain features of the data through data feature values, where the data feature values include the phase and amplitude of periodic data, as well as the mean and variance of aperiodic data; or extract the frequency - domain features of the data through data - processing methods, where the data - processing methods include Fourier transform, wavelet transform, and Laplace transform; finally, take each extracted time - domain or frequency - domain feature as each attribute of the tabular data and organize it into tabular data.

4. A data multi-dimensional anomaly detection system for implementing the data multi-dimensional anomaly detection method according to any one of claims 1 to 3, characterized in that: Including: An acquisition and processing module: used to acquire the original data, determine whether the original data is tabular data, and if it is not tabular data, convert the original data into tabular data; A pre - processing module: used to divide the tabular data into a training set and a test set and perform pre - processing on them separately; An improvement module: used to construct an auto - encoder and improve the loss function of the auto - encoder to achieve the conversion of the auto - encoder from completely reconstructing the input data to moderately reconstructing it; A training module: used to train the auto - encoder based on the improved loss function through the pre - processed training set to obtain a trained auto - encoder; A detection module: used to input the pre - processed test set into the trained auto - encoder to detect the outliers in the multi - dimensional tabular data.

5. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the method for detecting multi - dimensional anomalies of data as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the method for detecting multi - dimensional anomalies of data as described in any one of claims 1 to 3.

Images