Abnormal data detection model construction method and abnormal data detection system and method
By using kernel density estimation and logarithmic calculation loss functions in the automatic encoder, the problem of fewer labeled abnormal data samples is solved, and the ability to effectively detect abnormal data in a small amount of data is realized.
Patent Information
- Application Number
- CN202311286370.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-07
- Publication Date
- 2025-04-08
AI Technical Summary
The prior art is difficult to learn the characteristics of sample data when there are fewer exception data samples with labels, resulting in a degradation of model performance.
The data with exception tags are encoded and decoded using an autoencoder, potential characterization data are generated, and the model parameters iteratively updates using kernel density estimation and logarithmic calculation loss functions.
With a small number of abnormal data samples, the characteristics of the sample data can be effectively enlarged and a model that can detect abnormal data can be constructed.
Smart Images
Figure CN119782795A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of abnormal data detection, and specifically relates to an abnormal data detection model construction method, an abnormal data detection system and method. Background Art
[0002] With the development of informatization, the demand for information security protection is gradually increasing. Anomaly detection of information data is becoming more and more important. In practical applications, anomaly detection is an important research topic in many fields. The existence of abnormal data will not only affect security, but may also cause the performance of the model to decline, thereby affecting the effect and safety of practical applications. There are a large number of practical applications of abnormal data detection in many fields such as medicine, finance, manufacturing, transportation, etc. However, the number of abnormal data samples in real applications is very small. Due to the high cost of manual labeling, a large number of actual data labels are incomplete. When there are few labeled abnormal data samples, it is difficult for existing technologies to learn the characteristics of sample data. Summary of the invention
[0003] The purpose of the present invention is to solve the above-mentioned problems existing in the prior art and provide a method for constructing an abnormal data detection model, which can be trained when the number of abnormal data samples is small.
[0004] The present invention is achieved through the following technical solutions:
[0005] A method for building an abnormal data detection model includes the following steps:
[0006] Take the m-dimensional abnormal data with abnormal labels as training samples;
[0007] The training samples are input into the autoencoder, which encodes the reduced-dimensional latent representation data. After decoding, the latent representation data is reconstructed into m-dimensional data, and the model parameters of the autoencoder are iteratively updated using the loss function minimization principle.
[0008] The loss function is composed of two parts: the reconstruction error between the reconstructed data and its training samples is used as the first part of the loss function; the kernel density estimate of the potential representation is normalized and then logarithm is taken as the second part of the loss function.
[0009] Furthermore, the kernel density estimation formula is as follows:
[0010]
[0011] In the formula, h represents the sample bandwidth, which is used to control the width of the kernel function, k(x) represents the kernel function, N represents the number of sample data points, x represents the sample data point, and x i represents the i-th data point in the sample.
[0012] Furthermore, when the potential representation data distribution has obvious peaks or troughs, the rectangular kernel function is selected as the kernel function for kernel density estimation; when the overall trend of the potential representation data set needs to be estimated, the triangular kernel function is selected as the kernel function for kernel density estimation; when the edge of the potential representation data set needs to be estimated, the Epanechkov kernel function is selected as the kernel function for kernel density estimation.
[0013] Furthermore, a Gaussian kernel function is selected as the kernel function for kernel density estimation, and the kernel density formula is as follows:
[0014]
[0015] Where σ represents the bandwidth of the Gaussian kernel function.
[0016] Furthermore, when calculating the reconstruction error, the calculation method adopted includes calculating the mean square error, the mean absolute error, or the absolute value after subtraction.
[0017] The present invention also provides an abnormal data detection system, comprising an abnormal data detection model, wherein the abnormal data detection model is an automatic encoder trained by the abnormal data detection model construction method of the present invention.
[0018] The present invention also provides an abnormal data detection method, which includes inputting the data to be tested into the abnormal data detection system described in the present invention, and outputting reconstructed data; calculating the reconstruction error between the reconstructed data and the data to be tested, and when the reconstruction error is less than an error threshold, determining that the data to be tested is abnormal data.
[0019] Compared with the prior art, the beneficial effects of the present invention include:
[0020] 1. The loss function constructed by the present invention based on the potential representation data can amplify the characteristics of the sample data. Therefore, the present invention can be trained when there are fewer labeled abnormal data samples, so that the model can learn the characteristics of the sample data.
[0021] 2. The latent representation data is compressed in dimension as the sample data after dimensionality reduction. The kernel density estimation method is then used to extract the latent representation. The information extracted at this time is the data features of the amplified sample data. Using this information as a loss function to build a model, the model can be built when the amount of sample data is small.
[0022] 3. Calculate the data of each dimension of the potential representation data, normalize the calculation results and take the logarithm. This is done to ensure that the loss function value is minimized and the kernel density estimate of each dimension of the potential representation is maximized, thereby achieving the purpose of amplifying the characteristics of the sample data. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 Schematic diagram of the principle of adjusting model parameters through loss function during the training phase. DETAILED DESCRIPTION
[0024] The present invention is further described in detail below in conjunction with the accompanying drawings.
[0025] A method for building an abnormal data detection model includes the following steps:
[0026] Take the m-dimensional abnormal data with abnormal labels as training samples;
[0027] The training samples are input into the autoencoder, which encodes the reduced-dimensional latent representation data. After decoding, the latent representation data is reconstructed into m-dimensional data, and the model parameters of the autoencoder are iteratively updated using the loss function minimization principle.
[0028] The loss function is composed of two parts: the reconstruction error between the reconstructed data and its training samples is used as the first part of the loss function; the kernel density estimate of the potential representation is normalized and then logarithm is taken as the second part of the loss function.
[0029] The loss function is transformed to make it more suitable for anomaly detection in practical problems where only a small number of abnormal data samples exist.
[0030] The autoencoder will reconstruct the data, and often a threshold can be set based on the difference between the reconstructed data and the input data to filter out abnormal data.
[0031] During the encoding process of the autoencoder, the data will be reduced in dimension. The data after dimension reduction is called latent representation, which extracts the features of the original data to a certain extent. Taking advantage of this feature, the training data with labeled abnormal data as input is put into the autoencoder for training, and the latent representation after dimension reduction in the encoding stage is extracted again. After kernel density estimation and logarithmic calculation, it is finally fed back to the model as part of the model loss function together with the reconstruction error to build the model.
[0032] When calculating the reconstruction error, the calculation methods used include calculating the mean square error, the mean absolute error, or the absolute value after subtraction.
[0033] Example 1
[0034] Kernel density estimation is a non-parametric statistical method based on probability density function, which is often used to estimate the probability density function of unknown distribution. In KDE (Kernel Density Estimation), it is first assumed that the data sample is drawn from an unknown distribution, and it is hoped that the probability density function of the distribution can be inferred from the sample data. The core idea of KDE is to approximate the data sample points as independent and identically distributed random variables, and construct a kernel function with a fixed bandwidth at the position of each sample point.
[0035] The kernel density estimation formula is as follows:
[0036]
[0037] In the formula, h represents the sample bandwidth, which is used to control the width of the kernel function, k(x) represents the kernel function, N represents the number of sample data points, x represents the sample data point, and x i represents the i-th data point in the sample.
[0038] Usually, it can be represented by a Gaussian kernel function (normal distribution function), which is applicable to potential representation data of various distribution types. The mean of this Gaussian kernel function is the position of the sample point, and the variance is a pre-set fixed value. By superimposing multiple such kernel functions, a density estimation function of the data sample distribution is finally formed.
[0039] The formula of the Gaussian kernel function is as follows:
[0040]
[0041] This embodiment selects the Gaussian kernel function as the kernel function for kernel density estimation. The kernel density estimation formula is as follows:
[0042]
[0043] Where σ represents the bandwidth of the Gaussian kernel function.
[0044] refer to Figure 1 As shown in the figure, in the training stage of the anomaly detection model, the m-dimensional (m-number of features) abnormal data X with abnormal labels is input into the autoencoder. After the encoding process of the autoencoder, the reduced dimension data, that is, the n-dimensional potential representation data (the number of features after dimensionality reduction is n) is generated. The kernel density estimation value of the potential representation data in n dimensions is calculated, and the calculated result is normalized and the logarithm is taken as part of the loss function L r After the latent features are decoded, the m-dimensional reconstructed data X′ is obtained. The difference between the reconstructed data and the input data is calculated as the other part of the loss function L cThese two parts are connected to form an m+n-dimensional loss function, which is fed back to the model for training and the final training model is saved.
[0045] An abnormal data detection system includes an abnormal data detection model, wherein the abnormal data detection model is an autoencoder trained by the abnormal data detection model construction method of this embodiment.
[0046] An abnormal data detection method, inputting the data to be tested into the abnormal data detection system of this embodiment, outputting the reconstructed data; calculating the reconstruction error between the reconstructed data and the data to be tested, and when the reconstruction error is less than the error threshold, judging the data to be tested as abnormal data. When the reconstruction error is not less than the error threshold, judging the data to be tested as normal data.
[0047] Example 2
[0048] When performing kernel density estimation, a suitable kernel function can be selected according to the actual data, including but not limited to linear kernel function, polynomial kernel function, Gaussian kernel function, etc.
[0049] When the potential representation data distribution has obvious peaks or troughs, the rectangular kernel function is selected as the kernel function for kernel density estimation; when the overall trend of the potential representation data set needs to be estimated, the triangular kernel function is selected as the kernel function for kernel density estimation; when the edge of the potential representation data set needs to be estimated, the Epanechkov kernel function is selected as the kernel function for kernel density estimation.
[0050] An abnormal data detection system includes an abnormal data detection model, wherein the abnormal data detection model is an autoencoder trained by the abnormal data detection model construction method of this embodiment.
[0051] An abnormal data detection method, inputting the data to be tested into the abnormal data detection system of this embodiment, outputting the reconstructed data; calculating the reconstruction error between the reconstructed data and the data to be tested, and when the reconstruction error is less than the error threshold, judging the data to be tested as abnormal data. When the reconstruction error is not less than the error threshold, judging the data to be tested as normal data.
[0052] The above technical scheme is only a specific implementation method of the present invention. For those skilled in the art, it is easy to make various types of improvements or modifications based on the principles disclosed in the present invention, and it is not limited to the technical scheme described in the above specific embodiments of the present invention. Therefore, the above description is only preferred and does not have a restrictive meaning.
Claims
1. A method for constructing an abnormal data detection model, characterized in that: The following steps are involved: Take the m-dimensional abnormal data with abnormal labels as training samples; The training samples are input into the autoencoder, which encodes the reduced-dimensional latent representation data. After decoding, the latent representation data is reconstructed into m-dimensional data, and the model parameters of the autoencoder are iteratively updated using the loss function minimization principle. The loss function is composed of two parts: the reconstruction error between the reconstructed data and its training samples is used as the first part of the loss function; the kernel density estimate of the potential representation is normalized and then logarithm is taken as the second part of the loss function.
2. The method for constructing an abnormal data detection model according to claim 1, characterized in that: The kernel density estimation formula is as follows: In the formula, h represents the sample bandwidth, which is used to control the width of the kernel function, k(x) represents the kernel function, N represents the number of sample data points, x represents the sample data point, and x i represents the i-th data point in the sample.
3. The method for constructing an abnormal data detection model according to claim 1 or 2, characterized in that: When the potential representation data distribution has obvious peaks or troughs, the rectangular kernel function is selected as the kernel function for kernel density estimation; when the overall trend of the potential representation data set needs to be estimated, the triangular kernel function is selected as the kernel function for kernel density estimation; when the edge of the potential representation data set needs to be estimated, the Epanechkov kernel function is selected as the kernel function for kernel density estimation.
4. The method for constructing an abnormal data detection model according to claim 1 or 2, characterized in that: The Gaussian kernel function is selected as the kernel function of the kernel density estimation, and the kernel density estimation formula is as follows: Where σ represents the bandwidth of the Gaussian kernel function.
5. The method for constructing an abnormal data detection model according to claim 1, characterized in that: When calculating the reconstruction error, the calculation methods used include calculating the mean square error, the mean absolute error, or the absolute value after subtraction.
6. An abnormal data detection system, characterized in that: It comprises an abnormal data detection model, wherein the abnormal data detection model is an automatic encoder trained by the abnormal data detection model construction method according to any one of claims 1 to 5.
7. A method for detecting abnormal data, characterized in that: Input the data to be tested into the abnormal data detection system as claimed in claim 6, and output the reconstructed data; calculate the reconstruction error between the reconstructed data and the data to be tested, and when the reconstruction error is less than the error threshold, determine that the data to be tested is abnormal data.