A dynamic thresholded gaussian kernel density estimation system and method for sparse data

Abstract

This invention relates to a dynamic threshold Gaussian kernel density estimation system and method for sparse data. The method includes: preprocessing the original pcap data to extract useful information into a CSV file; then selecting an appropriate kernel function and bandwidth based on data characteristics to perform kernel density estimation; dynamically establishing a baseline for the distribution of login history status using the kernel density estimation results; and optimizing the dynamic threshold for the sparse characteristics of the login data to better adapt to the sparse data portion; finally, performing login anomaly detection based on the established baseline and threshold. Compared to traditional algorithms that do not consider data characteristics and do not establish dynamic thresholds, the estimation method of this invention significantly improves the anomaly detection rate on proprietary datasets.

Description

Technical Field

[0001] This invention relates to the field of baseline computation, and in particular to a dynamic threshold Gaussian kernel density estimation system and method for sparse data. Background Technology

[0002] Based on the availability of sufficient prior information, estimation methods can be mainly divided into parametric estimation methods and non-parametric estimation methods. Parametric estimation is a method of estimating unknown parameters in the population distribution based on a sample drawn from the population. People often need to analyze or infer the essential patterns reflected by the data they have. That is, how to choose statistics based on sample data to infer the distribution or numerical characteristics of the population.

[0003] In parametric estimation analysis, it is assumed that the data distribution conforms to a specific behavior, such as linear, transformable linear, or exponential, and then a specific solution is sought within the family of objective functions, i.e., the unknown parameters in the regression model are determined. In parametric discriminant analysis, it is assumed that the randomly selected data samples used as the basis for discrimination follow a specific distribution in each possible category. Experience and theory show that there is often a significant gap between these basic assumptions of parametric models and actual physical models, and these methods do not always yield satisfactory results.

[0004] In nonparametric estimation, the form of the population probability density function is unknown, requiring us to directly infer the probability density function itself. Some typical distribution forms commonly found in statistics do not always fit real-world distributions. Furthermore, many practical problems frequently encounter multimodal distributions, forcing us to infer the population distribution from a sample. Nonparametric estimation, also known as nonparametric density estimation, requires minimal prior knowledge, relies entirely on training data for estimation, and can be used for density estimation of arbitrary shapes. Common nonparametric estimation methods include histograms, kernel density estimation, and K-nearest neighbor estimation.

[0005] Existing baseline calculation algorithms mainly include polynomial fitting algorithms, probabilistic algorithms, ranking algorithms, wavelet theory, and neural network algorithms. They are primarily divided into two categories: static baseline algorithms and dynamic baseline algorithms. Static baselines are mainly set manually or automatically. Manual setting leverages the experience and knowledge of the setter, allowing for flexible and controllable adjustments. However, when the indicators fluctuate significantly, this method is susceptible to subjective influence and updates are not timely. Automatic setting methods can establish baselines based on the data's fluctuation range through simple statistics, but this method is difficult to maintain, lacks flexibility, and performs poorly with data exhibiting volatile changes. Existing dynamic baseline algorithms are broadly divided into two categories: dynamic baseline algorithms based on probabilistic methods and dynamic baseline algorithms based on ranking methods. However, a common problem with these algorithms is their failure to consider the characteristics of the samples, thus neglecting the potential for improvement in utilizing these characteristics to refine threshold processing. Summary of the Invention

[0006] To address the shortcomings of existing methods, this invention proposes a dynamic threshold Gaussian kernel density estimation system and method for sparse data. First, a dynamic baseline is established based on the Gaussian kernel density estimation results. Then, a dynamic threshold is set to address the sparse login log data points, thereby completing the detection of abnormal user logins.

[0007] The specific technical solution of the present invention is as follows:

[0008] A dynamic threshold Gaussian kernel density estimation system for sparse data includes a data collector and a processor; the data collector acquires data.

[0009] The processor preprocesses the raw pcap data, extracts useful information into a CSV file, and then selects an appropriate kernel function and bandwidth based on the data characteristics to perform kernel density estimation. The kernel density estimation results are used to dynamically establish a baseline for the distribution of login history status, and the dynamic threshold is optimized for the sparse characteristics of login data to better adapt to the sparse data part. Finally, login anomaly detection is performed based on the established baseline and threshold.

[0010] Furthermore, the collector extracts data from the pcap traffic packets.

[0011] Furthermore, preprocessing involves integrating the corresponding data packets into a stream, organizing the necessary parts into a CSV file format, and storing it for further use.

[0012] Furthermore, a Gaussian kernel is selected for kernel density estimation, and the bandwidth is chosen by minimizing the average integral squared error, calculated using the following formula:

[0013]

[0014] in, This represents the kernel density estimation result using bandwidth h, where f(x) represents the true value and E represents the mean.

[0015] Furthermore, the kernel density estimation specifically considers n independent and identically distributed sample points F, whose probability density function is f. The kernel density estimation is as follows:

[0016]

[0017] Where K(x) is the selected kernel function; h>0 is a smoothing parameter, which is the calculated bandwidth; K h (x) is the kernel function after bandwidth scaling.

[0018] Furthermore, the results of kernel density estimation prior to baseline establishment are sampled and exponentially restored, and a dynamic threshold is established for sparse data:

[0019] The pixel distribution model can be viewed as a weighted average of the contributions of all samples. The threshold Th is a critical value of the kernel density. First, sort the M sample points in ascending order, and then obtain x using the following method. begin and x end As shown in the following formula:

[0020]

[0021] Where D represents a constant, around 2.5, and an appropriate value is selected according to actual needs; h represents bandwidth; Xmax represents the maximum value in ascending order, and Xmin represents the minimum value;

[0022] x end Substituting into the kernel density formula, the result is:

[0023]

[0024] After obtaining the overall dynamic threshold, the threshold needs to be added to the dynamic probability baseline. The threshold added at each point is adjusted according to the normalized probability density of different points to optimize the threshold at sparse data points.

[0025] Furthermore, login anomaly detection is performed by estimating the kernel density of the user's data for the day based on the obtained historical login status baseline and threshold, and then comparing it with the historical baseline threshold.

[0026] This invention also relates to a dynamic threshold Gaussian kernel density estimation method for sparse data, comprising the following steps;

[0027] The original pcap data is preprocessed to extract useful information into a CSV file. Then, a suitable kernel function and bandwidth are selected based on the data characteristics to complete kernel density estimation. The kernel density estimation results are used to dynamically establish a baseline for the distribution of login history status, and the dynamic threshold is optimized for the sparse characteristics of login data to better adapt to the sparse data part. Finally, login anomaly detection is performed based on the established baseline and threshold.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] The dynamic threshold Gaussian kernel density estimation method for sparse data proposed in this invention was evaluated on our own dataset. Experiments show that the estimation method of this invention has a significantly improved anomaly detection rate on our own dataset compared with traditional algorithms that do not consider data characteristics and do not establish dynamic thresholds.

[0030] The method proposed in this invention fully considers the special status of sparse data in the entire sample set. Although the contribution of sparse data to the overall probability distribution estimation is very small, these regions have extremely low tolerance for thresholds. When establishing dynamic thresholds, it is necessary to consider the impact of the characteristics of this part of sparse data.

[0031] Finally, after verification, the dynamic threshold Gaussian kernel density estimation method of the present invention for sparse data has a better anomaly detection effect than the traditional method of estimating and establishing baseline thresholds when performing login anomaly detection tasks. It makes full use of the characteristics of the data itself and has stronger task adaptability. Attached Figure Description

[0032] Figure 1 This is a system block diagram of this embodiment;

[0033] Figure 2 These are diagrams showing the effects of different kernel functions;

[0034] Figure 3 This demonstrates the effects of different bandwidths;

[0035] Figure 4 It is a dynamic probability baseline plot;

[0036] Figure 5 It involves constructing a dynamic threshold graph. Detailed Implementation

[0037] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0038] Unless otherwise defined, the technical or scientific terms used in the embodiments of this application shall have the ordinary meaning understood by one of ordinary skill in the art. The terms "first," "second," and similar terms used in this embodiment do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two elements. Terms such as "upper," "lower," "left," "right," "horizontal," and "vertical" are used only relative to the orientation of the components in the accompanying drawings. These directional terms are relative concepts used for relative description and clarification, and they may change accordingly depending on the orientation of the components in the accompanying drawings.

[0039] like Figure 1 As shown, the dynamic threshold Gaussian kernel density estimation system for sparse data in this embodiment includes a collector 101, a processor 102, and a display 103; the collector 101 collects data.

[0040] Processor 102 preprocesses the raw pcap data, extracting useful information into a CSV file. Then, based on data characteristics, it selects an appropriate kernel function and bandwidth to perform kernel density estimation. Using the kernel density estimation results, it dynamically establishes a baseline for the distribution of login history states and optimizes the dynamic threshold to better suit the sparse data. Finally, it performs login anomaly detection based on the established baseline and threshold. Display 103 presents the final results.

[0041] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software through processing element calls; they can be implemented entirely in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware.

[0042] The processing element described herein can be an integrated circuit with signal processing capabilities. In implementation, each step or module of the above method can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0043] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to form a system-on-a-chip (SOC).

[0044] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a readable storage medium or transmitted from one readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0045] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0046] This embodiment utilizes the target distribution estimation capability of kernel density estimation to predict the target distribution and establish a baseline. Simultaneously, it performs dynamic threshold optimization to address the sparsity of the sample data, thus completing the user login anomaly detection task. Specifically, the raw pcap data is first preprocessed to extract useful information into a CSV file. Then, an appropriate kernel function and bandwidth are selected based on the data characteristics to perform kernel density estimation. Next, the kernel density estimation results are used to dynamically establish a baseline for the login history status distribution, and the dynamic threshold is optimized to better suit the sparse data portion. Finally, login anomaly detection is performed based on the established baseline and threshold.

[0047] Specifically, the dynamic threshold Gaussian kernel density estimation method for sparse data in this embodiment includes the following steps:

[0048] Step 1: Data preprocessing.

[0049] The raw data for this task differs from common login log files. Instead, it needs to be extracted from pcap traffic data packets. Therefore, the pcap file needs to be processed first to integrate the corresponding data packets into a stream. The parts we need should be organized into a CSV file and stored for later use.

[0050] Step 2: Select the kernel function.

[0051] A kernel function is a probability density function used in kernel density estimation. There are many types of kernel functions, each with its own strengths and weaknesses. Figure 2 This is the result achieved when using the same sample data to perform kernel density estimation using different kernel functions.

[0052] like Figure 2 As shown, even the same data can exhibit drastically different results when using different kernel functions. Figure 2 (1) is a histogram. Figure 2 (3) is a non-smooth kernel. Figure 2 (2, 4, 5, 6) represent smoothing kernels. In most cases, smoothing kernels are used in many scenarios. However, because the Gaussian kernel function has obvious statistical characteristics and is overly smooth overall, this embodiment chooses the Gaussian kernel for kernel density estimation.

[0053] Step 3: Select the appropriate bandwidth.

[0054] While different kernel functions can yield consistent conclusions—that is, the overall trend and density distribution patterns are generally consistent—kernel density functions are not perfect. Besides the choice of kernel algorithm, bandwidth also affects density estimation; excessively large or small bandwidth values ​​will negatively impact the estimation results. Figure 3 As shown, smaller bandwidth means higher sensitivity, but it also means a higher false alarm rate.

[0055] The same sample data and kernel function can result in significant differences when using different bandwidths. In this embodiment, bandwidth selection is performed by minimizing the average integral squared error, and the calculation formula is shown below:

[0056]

[0057] in, f(x) represents the calculated value using kernel density estimation with bandwidth h, f(x) represents the true value, and E represents the mean.

[0058] Step 4: Perform kernel density estimation.

[0059] After selecting a suitable kernel function and bandwidth, the next step is to perform kernel density estimation to calculate the probability distribution. Kernel density estimation is to use a smooth peak function (kernel function) to fit the observed data points, thereby simulating the true probability distribution curve.

[0060] Kernel density estimation is a nonparametric method for estimating probability density functions. Given n independent and identically distributed sample points F, with probability density function f, the kernel density estimate is as follows:

[0061]

[0062] Where K(x) is the kernel function selected in step 2. h>0 is a smoothing parameter, which is the bandwidth calculated in step 3. K h (x) is the kernel function after bandwidth scaling.

[0063] Step 5: Establish a dynamic baseline based on the estimated distribution.

[0064] Before establishing the baseline, we need to sample and exponentially restore the kernel density estimation results. The number of sampling points depends on the task requirements; in this example, the horizontal axis represents one day, so the number of sampling points is set to 2401.

[0065] Since the kernel density estimation in step 4 uses the sklearn function library in Python to calculate the log value of the probability corresponding to the sampling point, this embodiment performs exponential restoration on the kernel density estimation result in step 4, and finally plots the dynamic probability baseline result as shown below. Figure 4 As shown.

[0066] Step 6: Establish dynamic thresholds for sparse data.

[0067] Traditional fixed threshold strategies are very simple but not very sensitive to sparse data. In this embodiment, sparse data points are particularly easy to occur, but traditional fixed thresholds cannot detect this anomaly well. Therefore, a dynamic threshold construction method is used.

[0068] Assume the sample set contains M sample points, each sample x i Centered on itself, it makes a certain contribution to the overall distribution within a certain width range, x distance from the center. i The farther away, the smaller the contribution to the overall distribution. Therefore, the contribution function can actually be regarded as a function that is large in the middle and small at both ends.

[0069] Since adjacent sample points on the sample axis usually originate from the same local distribution, and since a Gaussian kernel function was used in step 2, we assume that this local distribution follows a Gaussian distribution, and the contribution function can be regarded as a Gaussian density function.

[0070] The pixel distribution can be viewed as a weighted sum of the contributions of all samples. The threshold Th is essentially a critical value for the kernel density. We first sort the M sample points in ascending order and then calculate x using the following method. begin and x end :

[0071]

[0072] Where D represents a constant, approximately 2.5, and a suitable value can be chosen according to actual needs; h represents the bandwidth; Xmax represents the maximum value in ascending order, and Xmin represents the minimum value. The reason for this is to treat the contribution of the samples as a Gaussian function. When sample point j is too far from sample point i, it is considered that the contribution of that sample point to the overall probability density is negligible, less than 0.0062. Therefore, x... begin and x end Substituting these values ​​into the kernel density formula yields the dynamic threshold Th, which is used in practice with x. begin and x endThe calculated probability densities are likely to be unequal because of the overall distribution, but the difference will not be too large. If x end Substituting into the kernel density formula, the result is:

[0073]

[0074] After obtaining the overall dynamic threshold, it needs to be added to the dynamic probability baseline. However, adding the same threshold to all points is obviously not the optimal solution, as it would make the threshold too high in areas of data sparsity. Therefore, this embodiment designs an approach that adjusts the threshold added at each point according to the normalized probability density of different points to optimize the threshold at data sparsity points. The final effect is as follows: Figure 5 As shown, a complete set of dynamic probability historical baselines that can be used for anomaly detection and thresholds optimized for sparse data were obtained.

[0075] Step 7: Perform login anomaly detection.

[0076] After the first six steps, the baseline and threshold of the user's historical login status have been obtained. Then, the login anomaly detection can be completed by estimating the kernel density of the user's data for the day according to the kernel function and bandwidth selected in steps 2 and 3, and comparing it with the historical baseline threshold.

[0077] That is, identify the portion of the day where the estimated kernel density baseline is greater than the dynamic threshold of the historical baseline, and if there is a login record in that portion, the login is judged as an abnormal login behavior.

[0078] Therefore, this embodiment addresses the problem of abnormal user login detection, which requires distribution estimation of experimental data and the establishment of a reasonable baseline. Based on the data characteristics and task objectives, this embodiment selects kernel density estimation, a nonparametric estimation method, for data distribution estimation and baseline establishment, and then sets the threshold accordingly.

[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic threshold Gaussian kernel density estimation system for sparse data, characterized in that: Includes a data collector and a processor; the data collector collects data; The processor preprocesses the raw pcap data, extracts useful information into a CSV file, and then selects an appropriate kernel function and bandwidth based on the data characteristics to perform kernel density estimation. The kernel density estimation results are used to dynamically establish a baseline for the distribution of login history status, and the dynamic threshold is optimized for the sparse characteristics of login data to better adapt to the sparse data part. Finally, login anomaly detection is performed based on the established baseline and threshold. The kernel density estimation results are sampled and exponentially restored, and a dynamic threshold is established for sparse data: The pixel distribution model can be viewed as a weighted average of the contributions of all samples. The threshold Th is a critical value of the kernel density. First, sort the M sample points in ascending order, and then obtain x using the following method. begin and x end As shown in the following formula: Where D represents a constant, taken as 2.5, and an appropriate value is selected according to actual needs; h represents bandwidth; Xmax represents the maximum value in ascending order, and Xmin represents the minimum value; x end Substituting into the kernel density formula, the result is: After obtaining the overall dynamic threshold, the threshold needs to be added to the dynamic probability baseline. The threshold added at each point is adjusted according to the normalized probability density of different points to optimize the threshold at sparse data points.

2. The system according to claim 1, characterized in that: The collector extracts data from the pcap traffic packets.

3. The system according to claim 1, characterized in that: Preprocessing involves merging the corresponding data packets into a stream, organizing the necessary parts into a CSV file format, and storing it for later use.

4. The system according to claim 1, characterized in that: A Gaussian kernel is used for kernel density estimation, and the bandwidth is selected by minimizing the average integral squared error, calculated using the following formula: ； in, This represents the kernel density estimation result using bandwidth h. E represents the true value, and E represents the mean.

5. The system according to claim 1, characterized in that: Specifically, kernel density estimation involves n independent and identically distributed sample points F, with probability density function f. The kernel density estimate is as follows: Where K(x) is the selected kernel function; h>0 is a smoothing parameter, which is the calculated bandwidth; K h (x) is the kernel function after bandwidth scaling.

6. The system according to claim 1, characterized in that: Login anomaly detection is achieved by estimating the kernel density of the user's current day's data based on the selected kernel function and bandwidth, and then comparing it with the historical baseline threshold.

7. A dynamic threshold Gaussian kernel density estimation method for sparse data, characterized in that: Includes the following steps; The original pcap data is preprocessed to extract useful information into a CSV file. Then, a suitable kernel function and bandwidth are selected based on the data characteristics to complete kernel density estimation. The kernel density estimation results are used to dynamically establish a baseline for the distribution of login history status, and the dynamic threshold is optimized for the sparse characteristics of login data to better adapt to the sparse data part. Finally, login anomaly detection is performed based on the established baseline and threshold. The kernel density estimation results are sampled and exponentially restored, and a dynamic threshold is established for sparse data: The pixel distribution model can be viewed as a weighted average of the contributions of all samples. The threshold Th is a critical value of the kernel density. First, sort the M sample points in ascending order, and then obtain x using the following method. begin and x end As shown in the following formula: Where D represents a constant, taken as 2.5, and an appropriate value is selected according to actual needs; h represents bandwidth; Xmax represents the maximum value in ascending order, and Xmin represents the minimum value; x end Substituting into the kernel density formula, the result is: After obtaining the overall dynamic threshold, the threshold needs to be added to the dynamic probability baseline. The threshold added at each point is adjusted according to the normalized probability density of different points to optimize the threshold at sparse data points.

Images