Outlier detection method, device, electronic device and storage medium

Abstract

The present invention discloses an outlier detection method, device, electronic device and storage medium. First, the local anomaly factor value of each data point in a data set is determined, and then the local anomaly factor value of each point is sorted in order of size. Based on the sorting result, a preset first number of first local anomaly factor values ​​with larger local anomaly factor values ​​are selected, and an outlier detection parameter value is determined according to a first difference between the selected first local anomaly factor value and the subsequent second local anomaly factor value, and then the outlier is determined according to the determined outlier detection parameter value. The solution provided by the embodiment of the present invention can determine the outlier detection parameter value of a data set suitable for outlier detection, thereby improving the rationality of the determined outliers.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an outlier detection method, device, electronic equipment and storage medium. Background Art

[0002] In some scenarios where clustering algorithms are used for behavior analysis, such as in the modeling process of password brute force attack detection, clustering methods are used to analyze the similarity of behavior patterns. When there are outliers, that is, outliers whose behavior is different from other patterns, the outliers will have an adverse effect on the clustering results, resulting in unstable behavior detection results. Therefore, outlier detection and filtering are required.

[0003] When performing outlier detection in the prior art, an assumption is generally made about the distribution of the data first, for example, the data is assumed to be Gaussian distributed, and the data outside a specific range of the distance mean is regarded as an outlier. In other words, the outlier detection algorithm in the prior art can be considered as a kind of estimated similarity. Whether it is through density, distance, angle or dividing hyperplane, it is necessary to specify the proportion and number of outliers or define a standard similarity reference value, and perform outlier detection based on the specified proportion and number of outliers or the defined similarity reference value. However, the specified proportion and number of outliers or the defined similarity reference value are all obtained based on experience or prior knowledge. The outliers determined in this way may deviate greatly from the actual situation, making it impossible to guarantee the rationality of the outliers. Summary of the invention

[0004] The embodiments of the present invention provide an outlier detection method, device, electronic device and storage medium, which are used to solve the problem of poor accuracy of existing outlier detection methods.

[0005] An embodiment of the present invention provides an outlier detection method, the method comprising:

[0006] Determine the local outlier factor value of each data point in the data set, and select a preset first number of first local outlier factor values ​​in descending order of the local outlier factor values;

[0007] For each first local anomaly factor value, determining a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​after the first local anomaly factor value; determining an outlier detection parameter value according to each determined first difference value;

[0008] A target local anomaly factor value is determined according to the outlier detection parameter value, and an outlier corresponding to the target local anomaly factor value is determined.

[0009] Further, determining the outlier detection parameter value according to each determined first difference value includes:

[0010] The minimum value of each determined first difference value is used as the outlier detection parameter value.

[0011] Furthermore, after selecting a preset first number of first local anomaly factor values, before determining, for each first local anomaly factor value, a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter, the method further includes:

[0012] Determine a discrete degree value according to each selected first local abnormal factor value, and judge whether the discrete degree value is less than a preset discrete threshold value;

[0013] If not, for each first local anomaly factor value, determining a difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter;

[0014] If yes, the method further comprises:

[0015] Make sure there are no outliers.

[0016] Furthermore, determining the discrete degree value according to each selected first local abnormal factor value includes:

[0017] The variance or standard deviation of each selected first local abnormal factor value is determined, and the variance or standard deviation is used as the discrete degree value.

[0018] Further, determining a target local anomaly factor value according to the outlier detection parameter value includes:

[0019] For each third local outlier factor value after the first local outlier factor value, determine the second difference between the third local outlier factor value and a preset third number of fourth local outlier factor values ​​thereafter; if the second difference is less than the outlier detection parameter value, determine the local outlier factor value before the third local outlier factor value as the target local outlier factor value.

[0020] Further, if the second difference is less than the outlier detection parameter value, determining the local anomaly factor value before the third local anomaly factor value as the target local anomaly factor value includes:

[0021] If the second difference is smaller than the outlier detection parameter value, the second difference values ​​corresponding to the fourth number of third local anomaly factor values ​​are successively preset after the third local anomaly factor value, and if each of the obtained second differences is smaller than the outlier detection parameter value, the local anomaly factor value before the third local anomaly factor value is determined as the target local anomaly factor value.

[0022] On the other hand, an embodiment of the present invention provides an outlier detection device, the device comprising:

[0023] A first determination module is used to determine the local anomaly factor value of each data point in the data set, and select a preset first number of first local anomaly factor values ​​in descending order of the local anomaly factor values;

[0024] A second determination module is used to determine, for each first local anomaly factor value, a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter; and determine an outlier detection parameter value according to each determined first difference;

[0025] The third determination module is used to determine the target local anomaly factor value according to the outlier detection parameter value, and determine the outliers corresponding to the target local anomaly factor value.

[0026] Furthermore, the second determination module is specifically configured to use the minimum value of each determined first difference as the outlier detection parameter value.

[0027] Furthermore, the device further comprises: a fourth determining module and a fifth determining module;

[0028] The fourth determination module is used to determine a discrete degree value according to each selected first local abnormal factor value, and judge whether the discrete degree value is less than a preset discrete threshold value; if not, trigger the second determination module, and if yes, trigger the fifth determination module;

[0029] The fifth determination module is used to determine whether there is no outlier.

[0030] Furthermore, the fourth determination module is specifically used to determine the variance or standard deviation of each selected first local abnormal factor value, and use the variance or standard deviation as the discrete degree value.

[0031] Furthermore, the third determination module is specifically used to determine, in sequence, for each third local anomaly factor value after the first local anomaly factor value, a second difference between the third local anomaly factor value and a preset third number of fourth local anomaly factor values ​​thereafter; if the second difference is less than the outlier detection parameter value, then determine that the local anomaly factor value before the third local anomaly factor value is the target local anomaly factor value.

[0032] Furthermore, the third determination module is specifically used to determine, if the second difference is smaller than the outlier detection parameter value, the second difference values ​​corresponding to the third local anomaly factor values ​​respectively preset for a fourth number after the third local anomaly factor value in sequence; if each of the obtained second differences is smaller than the outlier detection parameter value, determine the local anomaly factor value before the third local anomaly factor value as the target local anomaly factor value.

[0033] In another aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0034] Memory, used to store computer programs;

[0035] The processor is used to implement any of the method steps described above when executing the program stored in the memory.

[0036] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of the above are implemented.

[0037] An embodiment of the present invention provides an outlier detection method, device, electronic device and storage medium, the method comprising: determining a local anomaly factor value for each data point in a data set, and selecting a preset first number of first local anomaly factor values ​​in descending order of the local anomaly factor values; for each first local anomaly factor value, determining a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter; determining an outlier detection parameter value according to each determined first difference; determining a target local anomaly factor value according to the outlier detection parameter value, and determining the outliers corresponding to the target local anomaly factor value.

[0038] The above technical solution has the following advantages or beneficial effects:

[0039] In an embodiment of the present invention, the local anomaly factor value of each data point in the data set is first determined, and then the local anomaly factor value of each point is sorted in order of size. Based on the sorting result, a first number of first local anomaly factor values ​​with a preset larger local anomaly factor value are selected, and an outlier detection parameter value is determined according to a first difference between the selected first local anomaly factor value and the subsequent second local anomaly factor value, and then the outliers are determined according to the determined outlier detection parameter value. The present application does not need to specify the proportion, number or defined similarity reference value of the outliers, and adaptively determines the outlier detection parameter value corresponding to the data set for outlier detection, and then determines the outliers in the data set according to the determined outlier detection parameter value. The scheme provided in the embodiment of the present invention can determine the outlier detection parameter value of the data set suitable for outlier detection, thereby improving the rationality of the determined outliers. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A schematic diagram of an outlier detection process provided in Example 1 of the present invention;

[0042] Figure 2 A schematic diagram of an outlier detection process provided in Example 3 of the present invention;

[0043] Figure 3 A schematic diagram of an application scenario provided by Embodiment 3 of the present invention;

[0044] Figure 4 A schematic diagram of the structure of an outlier detection device provided in Example 4 of the present invention;

[0045] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Example 5 of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Embodiment 1:

[0048] Figure 1A schematic diagram of an outlier detection process provided by an embodiment of the present invention, the process includes the following steps:

[0049] S101: Determine the local outlier factor value of each data point in the data set, and select a preset first number of first local outlier factor values ​​in descending order of the local outlier factor values.

[0050] S102: for each first local anomaly factor value, determining a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter; and determining an outlier detection parameter value according to each determined first difference.

[0051] S103: Determine a target local anomaly factor value according to the outlier detection parameter value, and determine the outliers corresponding to the target local anomaly factor value.

[0052] The outlier detection method provided by the embodiment of the present invention is applied to an electronic device, which may be a device such as a PC, a tablet computer, or a server.

[0053] The electronic device determines the local outlier factor value of each data point in the data set. Taking the detection of password blasting attack behavior as an example, the data set is each log obtained. Among them, each log generally has four field attributes, namely login time, login IP address, login user name and login result, and the login result includes two results: success and failure. Feature extraction is performed on each log to obtain the feature vector corresponding to each log. The feature vector corresponding to each log can be understood as each data point in the data set in the embodiment of the present invention. The local outlier factor value of each data point in the data set is determined based on the local outlier factor algorithm (Local Outlier Factor, LOF), and then each local outlier factor value is sorted in order from large to small.

[0054] According to the sorting result, a preset first number of first local anomaly factor values ​​are selected from front to back, wherein the preset first number may be 10, 15, 20, etc. For each first local anomaly factor value, a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter is determined, wherein the preset second number may be the same as or different from the preset first number, and the preset second number may be 10, 12, 15, etc.

[0055] For example, the local anomaly factor value of each data point in the data set is sorted in descending order according to the local anomaly factor value, and is 2.0, 1.8, 1.75, 1.7, 1.3, 1.2, 1.15, 1.1, 1, 0.9, 0.85, 0.7, 0.6, 0.55, 0.54. The preset first number is 3, and the preset second number is 5. Then, the first local anomaly factor values ​​of the preset first number are selected as 2.0, 1.8, and 1.75. For each first local anomaly factor value, determine the first difference between the first local anomaly factor value and the second local anomaly factor value of the preset second number of intervals thereafter. Determine the first difference between the first local anomaly factor value 2.0 and the second local anomaly factor value 1.15 of the preset second number of intervals thereafter, which is 2.0-1.15=0.85. Determine the first difference between the first local anomaly factor value 1.8 and the second local anomaly factor value 1.1 of the preset second number of intervals thereafter, which is 1.8-1.1=0.7. A first difference value 1.75-1=0.75 between the first local anomaly factor value 1.75 and a second number of second local anomaly factor values ​​1 preset thereafter is determined.

[0056] The outlier detection parameter value is determined according to each determined first difference, wherein, after determining each first difference, the average value of each first difference is optionally used as the outlier detection parameter value, the second minimum value or the third minimum value in each first difference is optionally used as the outlier detection parameter value, a set number of smaller first differences are optionally selected, and then the average value of the selected set number of smaller first differences is used as the outlier detection parameter value, etc. Preferably, the minimum value in each determined first difference can be used as the outlier detection parameter value. The above example is also used for explanation. After determining the three first difference values ​​of 0.85, 0.7 and 0.75, the average value of the three can be calculated, (0.85+0.7+0.75) / 3=0.77 can be used as the outlier detection parameter value, the second minimum value of 0.75 can be used as the outlier detection parameter value, etc. Preferably, the smallest first difference value of 0.7 can be selected as the outlier detection parameter value.

[0057] After determining the outlier detection parameter value, the target local anomaly factor value is determined according to the outlier detection parameter value. For example, for the local anomaly factor value of each data point in the data set, the difference between the local anomaly factor value and the local anomaly factor values ​​of a preset number of intervals thereafter is determined. If the obtained difference is greater than the outlier detection parameter value, the local anomaly factor value is determined to be the target local anomaly factor value. Then, the data point corresponding to the target local anomaly factor value is determined as the outlier in the data set.

[0058] In an embodiment of the present invention, the local anomaly factor value of each data point in the data set is first determined, and then the local anomaly factor value of each point is sorted in order of size. Based on the sorting result, a first number of first local anomaly factor values ​​with a preset larger local anomaly factor value are selected, and an outlier detection parameter value is determined according to a first difference between the selected first local anomaly factor value and the subsequent second local anomaly factor value, and then the outliers are determined according to the determined outlier detection parameter value. The present application does not need to specify the proportion, number or defined similarity reference value of the outliers, and adaptively determines the outlier detection parameter value corresponding to the data set for outlier detection, and then determines the outliers in the data set according to the determined outlier detection parameter value. The scheme provided in the embodiment of the present invention can determine the outlier detection parameter value of the data set suitable for outlier detection, thereby improving the rationality of the determined outliers.

[0059] Embodiment 2:

[0060] In order to improve processing efficiency, based on the above embodiment, in an embodiment of the present invention, after selecting a preset first number of first local anomaly factor values, before determining, for each first local anomaly factor value, a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter, the method further includes:

[0061] Determine a discrete degree value according to each selected first local abnormal factor value, and judge whether the discrete degree value is less than a preset discrete threshold value;

[0062] If not, for each first local anomaly factor value, determining a difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter;

[0063] If yes, the method further comprises:

[0064] Make sure there are no outliers.

[0065] In the embodiment of the present invention, after the electronic device selects a preset first number of first local anomaly factor values, it first determines the discreteness value according to each selected first local anomaly factor value. If the discreteness value is small, it means that the clustering similarity of the data points corresponding to the larger local anomaly factor value is very high, and it is determined that there are no outliers in the data set. If the discreteness value is large, it means that the clustering similarity of the data points corresponding to the larger local anomaly factor value is low, and then the process of determining the first difference between the first local anomaly factor value and the second local anomaly factor value preset in the interval after it is performed for each first local anomaly factor value, and then determining the outlier.

[0066] Specifically, a preset discrete threshold is stored in the electronic device, and the preset discrete threshold may be a value close to 0, such as 0.1, 0.05, etc. After determining the discrete degree value of each selected first local abnormal factor value, it is determined whether the discrete degree value is less than the preset discrete threshold, and if so, it is determined that there is no outlier, otherwise, the subsequent process of determining the outlier is performed.

[0067] According to each selected first local abnormal factor value, the discrete degree value is determined. Optionally, the range, quartile, mean difference, heterogeneity ratio, coefficient of dispersion and other statistics of each selected first local abnormal factor value are determined as the discrete degree value. Preferably, the variance or standard deviation of each selected first local abnormal factor value is determined, and the variance or standard deviation is used as the discrete degree value.

[0068] Because in the embodiment of the present invention, before determining, for each first local anomaly factor value, the first difference between the first local anomaly factor value and the second number of second local anomaly factor values ​​preset at an interval thereafter, the discrete degree value is first determined, if it is determined according to the discrete degree value that there are no outliers, the subsequent process of determining the outliers is omitted, and when it is determined according to the discrete degree value that there are outliers, the subsequent process is performed, thereby improving the overall data processing efficiency.

[0069] Embodiment 3:

[0070] In order to make the determination of the target local anomaly factor value more accurate, and thus make the outlier detection more accurate, based on the above embodiments, in an embodiment of the present invention, the determination of the target local anomaly factor value according to the outlier detection parameter value includes:

[0071] For each third local outlier factor value after the first local outlier factor value, determine the second difference between the third local outlier factor value and a preset third number of fourth local outlier factor values ​​thereafter; if the second difference is less than the outlier detection parameter value, determine the local outlier factor value before the third local outlier factor value as the target local outlier factor value.

[0072] The preset third number and the preset second number may be the same or different. For each third local abnormal factor value after the first local abnormal factor value, after determining the second difference between the third local abnormal factor value and the fourth local abnormal factor value with a preset third number after it, it is determined whether the second difference is less than the outlier detection parameter value. If not, the second difference between the next third local abnormal factor value and the fourth local abnormal factor value with a preset third number after it is calculated. If yes, all local abnormal factor values ​​before the third local abnormal factor value are determined as target local abnormal factor values.

[0073] For example, the local anomaly factor value of each data point in the data set is sorted in descending order according to the local anomaly factor value, and is 2.0, 1.8, 1.75, 1.7, 1.3, 1.2, 1.15, 1.1, 1, 0.9, 0.85, 0.7, 0.6, 0.55, and 0.54. The preset first number is 3, the preset second number is 5, and the preset third number is 5. Then, the first local anomaly factor values ​​of the preset first number are selected as 2.0, 1.8, and 1.75, respectively. For each first local anomaly factor value, determine the first difference between the first local anomaly factor value and the second local anomaly factor value of the preset second number after it. Determine three first difference values ​​of 0.85, 0.7, and 0.75, and select the smallest first difference value of 0.7 as the outlier detection parameter value. At this time, for each third local anomaly factor value after the first local anomaly factor value, the second difference between the third local anomaly factor value and the third number of fourth local anomaly factor values ​​after it is determined in turn, that is, the second difference between the third local anomaly factor value 1.7 after the first local anomaly factor value and the third number of fourth local anomaly factor values ​​0.9 after it is calculated as 1.7-0.9=0.8. If 0.8 is not less than the outlier detection parameter value 0.7, then continue to calculate 1.3-0.85=0.45. If 0.45 is less than the outlier detection parameter value 0.7, then all local anomaly factor values ​​2.0, 1.8, 1.75, and 1.7 before the third local anomaly factor value 1.3 are determined as target local anomaly factor values.

[0074] In order to further determine the target local anomaly factor value more accurately, in an embodiment of the present invention, if the second difference is less than the outlier detection parameter value, determining the local anomaly factor value before the third local anomaly factor value as the target local anomaly factor value includes:

[0075] If the second difference is smaller than the outlier detection parameter value, the second difference values ​​corresponding to the fourth number of third local anomaly factor values ​​are successively preset after the third local anomaly factor value, and if each of the obtained second differences is smaller than the outlier detection parameter value, the local anomaly factor value before the third local anomaly factor value is determined as the target local anomaly factor value.

[0076] Among them, the preset fourth number can be 2, 3, etc. The above example is also used for illustration. For example, the local anomaly factor value of each data point in the data set is 2.0, 1.8, 1.75, 1.7, 1.3, 1.2, 1.15, 1.1, 1, 0.9, 0.85, 0.7, 0.6, 0.55, and 0.54 respectively after being sorted in order from large to small according to the local anomaly factor value. The preset first number is 3, the preset second number is 5, the preset third number is 5, and the preset fourth number is 2, then the first local anomaly factor values ​​of the preset first number are selected as 2.0, 1.8, and 1.75 respectively. For each first local anomaly factor value, determine the first difference between the first local anomaly factor value and the second local anomaly factor value of the preset second number after it. Determine three first difference values ​​0.85, 0.7, and 0.75, and select the smallest first difference value 0.7 as the outlier detection parameter value. At this time, for each third local anomaly factor value after the first local anomaly factor value, the second difference between the third local anomaly factor value and the third number of fourth local anomaly factor values ​​after it is determined in turn, that is, the second difference 1.7-0.9=0.8 between the third local anomaly factor value 1.7 after the first local anomaly factor value and the third number of fourth local anomaly factor values ​​0.9 after it is calculated. If 0.8 is not less than the outlier detection parameter value 0.7, then continue to calculate 1.3-0.85=0.45. If 0.45 is less than the outlier detection parameter value 0.7, then continue to calculate 1.2-0.7=0.5. If 0.5 is less than the outlier detection parameter value 0.7, then all local anomaly factor values ​​2.0, 1.8, 1.75, and 1.7 before the third local anomaly factor value 1.3 are determined as target local anomaly factor values.

[0077] In addition, it should be noted that the outlier detection parameter value can be adjusted according to the actual scenario. For example, after the outlier detection parameter value is determined, the outlier detection parameter value can be reduced or enlarged as needed, and then outlier detection is performed based on the reduced or enlarged outlier detection parameter value.

[0078] Since in the embodiment of the present invention, after determining the outlier detection parameter value, if the second difference values ​​corresponding to the fourth number of third local anomaly factor values ​​preset continuously after the first local anomaly factor value are all smaller than the outlier detection parameter value, the local anomaly factor value before the fourth number of third local anomaly factor values ​​preset continuously is determined as the target local anomaly factor value. Thus, the determined target local anomaly factor value is more adaptive, and thus the rationality of outlier detection is higher.

[0079] Figure 2 The schematic diagram of the outlier detection process provided by the embodiment of the present invention includes:

[0080] S201: Determine the local outlier factor value of each data point in the data set, and select a preset first number of first local outlier factor values ​​in descending order of the local outlier factor values.

[0081] S202: Determine a discrete degree value according to each selected first local abnormal factor value, and judge whether the discrete degree value is less than a preset discrete threshold value. If yes, determine that there is no outlier. If not, proceed to S203.

[0082] S203: for each first local anomaly factor value, determine a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter, and use the minimum value of each determined first difference as an outlier detection parameter value.

[0083] S204: determining, for each third local outlier factor value after the first local outlier factor value, a second difference between the third local outlier factor value and a preset third number of fourth local outlier factor values ​​thereafter; if the second difference is less than the outlier detection parameter value, determining the local outlier factor value before the third local outlier factor value as the target local outlier factor value, and determining the outlier corresponding to the target local outlier factor value.

[0084] The outlier detection method provided by the embodiment of the present invention is described in detail below in conjunction with specific scenarios.

[0085] For outlier detection, there are usually two types of methods in related technologies:

[0086] Statistics and probability models: mainly make assumptions about data distribution and find "anomalies" under the assumptions, often using extreme value analysis or hypothesis testing. For example, one-dimensional data assumes a Gaussian distribution and treats data outside a certain range from the mean as anomalies.

[0087] Models based on similarity measurement: The distribution of abnormal points and normal points is different. For example, in the K-NearestNeighbor (KNN) algorithm, the distance between a sample and its kth nearest neighbor is considered an abnormal value, and the k-nearest neighbor distance of the abnormal point is larger. LOF, determined by local data density. Isolation Forest (ISF), by dividing the hyperplane, calculates the number of hyperplanes required to "isolate" a sample. In a low-density space, the number of divisions required to isolate a sample is less. PCA, by finding k eigenvectors and calculating the reconstruction error of each sample after being projected by these k eigenvectors, the reconstruction error of normal points should be smaller.

[0088] In other words, most anomaly detection algorithms can be thought of as estimating similarity, whether through density, distance, angle, or dividing hyperplane.

[0089] The defects of the related art are:

[0090] (1) It is necessary to obtain the number, proportion, number of categories, etc. of abnormal events as prior knowledge in advance, and when the actual situation is inconsistent with the previous distribution, large errors are likely to occur.

[0091] (2) Regardless of density, distance, angle, or partitioning hyperplane estimation, a standard similarity reference value needs to be defined. When the dimension is too high, this similarity or value is very abstract and sensitive, difficult to estimate, and can only be adjusted repeatedly in multiple experiments. Similarly, when the data distribution is inconsistent with the previous distribution, it will also cause large errors.

[0092] In summary, the evaluation parameters of existing methods are too sensitive to the clustering results, or difficult to judge using intuitive methods, so it is impossible to achieve better model (algorithm) robustness.

[0093] The embodiment of the present invention provides an outlier detection method based on density distribution, which adds a second-order calculation based on factor values ​​on the basis of using an abnormal factor detection algorithm, and obtains points with a larger degree of outlier distribution as outliers.

[0094] The outlier detection method provided by the embodiment of the present invention does not need to specify the abnormal proportion or number: the ratio interval of the density difference is calculated based on the second-order density value, thereby determining the outlier. There are no highly sensitive parameters and the parameters are easy to understand: the parameters are the benchmark number and the difference ratio, etc., which are easier to understand intuitively than the process quantities in the algorithm such as the clustering distance. When the parameters are changed, the impact on the results is also small.

[0095] The specific process is as follows:

[0096] First, the local anomaly factor value of each data point in the data set is calculated based on the density. The specific formula is as follows:

[0097]

[0098] The above formula represents the neighborhood point N of point p k The average ratio of the local reachability density of point (p) to the local reachability density of point p. k (p) represents the local reachable density of point p. When the ratio is closer to 1, it means that the density of its neighborhood points is similar.

[0099] According to the local anomaly factor values, the values ​​are sorted from large to small, and the discrete degree values ​​of the first local anomaly factor values ​​preset in the first quantity are calculated. The discrete degree values ​​can be calculated according to the variance or the standard deviation, such as shown in the following formula:

[0100]

[0101] Among them, σ 2 is the population variance, X is the variable, μ is the population mean, and N is the number of cases.

[0102] If d is close to 0, that is, the discrete degree value is less than the preset discrete threshold, it means that there is no outlier. Otherwise, the difference between the first local anomaly factor value of the preset first number and the second local anomaly factor value of the preset second number is calculated, and the minimum value of the difference is used as the reference value for outlier detection:

[0103] r * =min{LOF(i)-LOF(i+n)|i∈1, 2...k};

[0104] Here, k represents the LOF value difference of the first k local outlier factor values ​​before calculation, that is, k is a preset first number.

[0105] The second difference value corresponding to the third local anomaly factor value after the first local anomaly factor value is calculated sequentially:

[0106] r j =LOF(j)-LOF(j+n);

[0107] Wherein, rj represents the LOF value difference corresponding to the jth third local outlier factor value.

[0108] Compare the LOF value differences. When rj<=r*, the data points corresponding to the local outlier factor values ​​before the third local outlier factor value j are determined as outliers. Otherwise, the LOF value difference corresponding to the third local outlier factor value j+1 is calculated.

[0109] The outlier detection method provided by the embodiment of the present invention can be applied in the following scenarios: based on logs or traffic, a clustering model for user login behavior analysis is established to find event streams or event clusters with similar behavior patterns. For example, when detecting password blasting attack behavior, since blasting events do not have a standard statistical distribution (may occur several times a day, or may occur 0 times), they cannot be directly eliminated in proportion. If outlier filtering is not performed, it will cause great interference to the accuracy of the clustering algorithm (multiple outliers will affect the density clustering effect). Therefore, the embodiment of the present invention is based on second-order density, and uses points with large density changes as boundary points to distinguish outliers from ordinary points.

[0110] Below Figure 3 Taking the scenario shown in the figure as an example, the process of outlier detection is explained:

[0111] 1. Read logs: Read each line of logs in turn. Assume that each line of logs has four field attributes, namely login time, login IP, login username, and login result.

[0112] 2. Data cleaning: Select the logs that are fully recorded and whose login results are failure.

[0113] 3. Feature extraction: For each row of logs with a login result of fail, perform 128-dimensional data feature extraction. During feature extraction, all logs including success and failure must be included.

[0114] 4. Data regularization: For the feature vector set after feature extraction, fill in the empty values ​​of the data, standardize the data, and normalize the data.

[0115] 5. Feature dimensionality reduction: Since the number of failed login logs in a cycle is less than 1,000, the 128-dimensional feature set will be reduced to 16 dimensions based on the principal component analysis (PCA) algorithm.

[0116] 6. Pre-selection of outliers: Based on LOF, extract 20% of the total amount as preliminary screening candidate points, and select the first local anomaly factor values ​​of the preset first number.

[0117] 7. Second-order density outlier detection: Perform second-order anomaly detection based on density to obtain outliers.

[0118] 8. Density clustering: Based on the DBSCAN clustering algorithm, the minimum number of samples is set to 5 according to experience (at least 5 visits are defined as this type of behavior pattern). The grid search method is used to search for the eps value, and then the maximum value of the Calinski-Harabasz score is used as the basis for the value selection. The optimal eps value is automatically selected to complete the clustering.

[0119] 9. Result output: Save each type of result, where -1 represents an abnormal category, that is, a normal login failure log, and other numbers represent category numbers.

[0120] For step 7, a specific example of the algorithm is given: assuming that after LOF, 15 points are initially screened, and the factor values ​​after sorting from large to small are as follows (if this ratio is closer to 1, it means that the density of p and its neighborhood points is similar, and p may belong to the same cluster as its neighborhood; if this ratio is less than 1, it means that the density of p is higher than the density of its neighborhood points, and p is a dense point; if this ratio is greater than 1, it means that the density of p is less than the density of its neighborhood points, and p is more likely to be an outlier).

[0121] 2.0, 1.8, 1.75, 1.7, 1.3, 1.2, 1.15, 1.1, 1, 0.9, 0.85, 0.7, 0.6, 0.55, 0.54.

[0122] Assume the variance value interval is 5, the variance threshold is 0.01, k=3, n=5, and the tolerance interval is 10%;

[0123] (1) Calculate the variance:

[0124] The variance of the first 5 calculated values ​​= 0.0524>0.01, indicating that the outliers should continue to be calculated.

[0125] (2) Calculate r * :

[0126] Since k=3, only the first 3 r values ​​are calculated:

[0127] r1=LOF1-LOF6=2-1.2=0.8;

[0128] r2=LOF2-LOF7=1.8-1.15=0.65;

[0129] r3=LOF3-LOF8=1.75-1.1=0.65;

[0130] Therefore, r*=min{r1, r2, r3}=0.65.

[0131] (3) Calculate r in sequence and compare it with r* until the condition is no longer met:

[0132] r4=LOF4-LOF9=1.7-1=0.7>r*;

[0133] r5=LOF5-LOF10=1.3-0.9=0.4<(1-10%)*r*;

[0134] r6=LOF5-LOF11=1.2-0.85=0.35<(1-10%)*r*;

[0135] (4) Therefore, the first four points are determined to be outliers.

[0136] The necessity analysis of the embodiments of the present invention is as follows:

[0137] (1) Behavior patterns cannot be predicted based on existing attack events. For example, the probability of password cracking in previous events is 1‰, but the proportion of abnormal point filtering cannot be defined as 1‰ or other specific proportions, because future attack events are independent events and do not rely on prior knowledge.

[0138] (2) After PCA, the feature dimension is reduced to 16 dimensions, the original feature meaning is lost, and it is unclear how to define the similarity of clustering. Taking K nearest neighbor as an example, it is impossible to know whether 0.01 or 0.001 is the appropriate outlier classification boundary, and the algorithm parameter adjustment requires a lot of time and experience, making it difficult to apply.

[0139] Embodiment 4:

[0140] Figure 4 A schematic diagram of the structure of an outlier detection device provided in an embodiment of the present invention, the device comprises:

[0141] A first determination module 41 is used to determine the local anomaly factor value of each data point in the data set, and select a preset first number of first local anomaly factor values ​​in descending order of the local anomaly factor values;

[0142] A second determination module 42 is used to determine, for each first local anomaly factor value, a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter; and determine an outlier detection parameter value according to each determined first difference;

[0143] The third determination module 43 is used to determine the target local anomaly factor value according to the outlier detection parameter value, and determine the outliers corresponding to the target local anomaly factor value.

[0144] The second determination module 42 is specifically configured to use the minimum value of each determined first difference as the outlier detection parameter value.

[0145] The device further includes: a fourth determination module 44 and a fifth determination module 45;

[0146] The fourth determination module 44 is used to determine the discrete degree value according to each selected first local abnormal factor value, and judge whether the discrete degree value is less than a preset discrete threshold value; if not, trigger the second determination module 42, and if yes, trigger the fifth determination module 45;

[0147] The fifth determining module 45 is used to determine that there is no outlier.

[0148] The fourth determination module 44 is specifically configured to determine the variance or standard deviation of each selected first local abnormal factor value, and use the variance or standard deviation as the discrete degree value.

[0149] The third determination module 43 is specifically used to determine, for each third local anomaly factor value after the first local anomaly factor value, a second difference between the third local anomaly factor value and a preset third number of fourth local anomaly factor values ​​thereafter; if the second difference is less than the outlier detection parameter value, determine the local anomaly factor value before the third local anomaly factor value as the target local anomaly factor value.

[0150] The third determination module 43 is specifically used to determine, if the second difference is smaller than the outlier detection parameter value, the second difference values ​​corresponding to the third local anomaly factor values ​​respectively preset in succession after the third local anomaly factor value, and if each of the obtained second differences is smaller than the outlier detection parameter value, determine the local anomaly factor value before the third local anomaly factor value as the target local anomaly factor value.

[0151] Embodiment 5:

[0152] Based on the above embodiments, an electronic device is also provided in an embodiment of the present invention, such as Figure 5 As shown, it includes: a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304;

[0153] The memory 303 stores a computer program. When the program is executed by the processor 301, the processor 301 performs the following steps:

[0154] Determine the local outlier factor value of each data point in the data set, and select a preset first number of first local outlier factor values ​​in descending order of the local outlier factor values;

[0155] For each first local anomaly factor value, determining a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​after the first local anomaly factor value; determining an outlier detection parameter value according to each determined first difference value;

[0156] A target local anomaly factor value is determined according to the outlier detection parameter value, and an outlier corresponding to the target local anomaly factor value is determined.

[0157] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present invention. Since the principle of solving the problem by the above electronic device is similar to that of the outlier detection method, the implementation of the above electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0158] The electronic device provided in the embodiment of the present invention may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (PDA), a network side device, etc.

[0159] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0160] The communication interface 302 is used for communication between the electronic device and other devices.

[0161] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0162] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0163] Embodiment 6:

[0164] On the basis of the above embodiments, an embodiment of the present invention further provides a computer storage readable storage medium, wherein the computer readable storage medium stores a computer program executable by an electronic device, and when the program is run on the electronic device, the electronic device implements the following steps when executing:

[0165] Determine the local outlier factor value of each data point in the data set, and select a preset first number of first local outlier factor values ​​in descending order of the local outlier factor values;

[0166] For each first local anomaly factor value, determining a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​after the first local anomaly factor value; determining an outlier detection parameter value according to each determined first difference value;

[0167] A target local anomaly factor value is determined according to the outlier detection parameter value, and an outlier corresponding to the target local anomaly factor value is determined.

[0168] Based on the same inventive concept, a computer-readable storage medium is also provided in an embodiment of the present invention. Since the principle of solving the problem when the processor executes the computer program stored on the above-mentioned computer-readable storage medium is similar to that of the outlier detection method, the implementation of the processor when executing the computer program stored on the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0169] The above-mentioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor in the electronic device, including but not limited to magnetic storage such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc., optical storage such as CD, DVD, BD, HVD, etc., and semiconductor storage such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD), etc.

[0170] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0173] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0174] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for detecting outliers, characterized in that: The method comprises: In the process of detecting password brute force attack behavior, each acquired log is used to form a data set, and feature extraction is performed on each log to obtain a feature vector corresponding to each log, and the feature vector corresponding to each log is each data point in the data set; the local anomaly factor value of each data point in the data set is determined, and a preset first number of first local anomaly factor values ​​are selected in descending order of the local anomaly factor values; For each first local anomaly factor value, determining a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​after the first local anomaly factor value; determining an outlier detection parameter value according to each determined first difference value; Determine a target local anomaly factor value according to the outlier detection parameter value, and determine the outlier corresponding to the target local anomaly factor value; Determining the target local anomaly factor value according to the outlier detection parameter value comprises: For each third local outlier factor value after the first local outlier factor value, determine the second difference between the third local outlier factor value and a preset third number of fourth local outlier factor values ​​thereafter; if the second difference is less than the outlier detection parameter value, determine the local outlier factor value before the third local outlier factor value as the target local outlier factor value.

2. The method according to claim 1, characterized in that Determining the outlier detection parameter value according to each determined first difference value comprises: The minimum value of each determined first difference value is used as the outlier detection parameter value.

3. The method according to claim 1, characterized in that After selecting a preset first number of first local anomaly factor values, and before determining, for each first local anomaly factor value, a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter, the method further comprises: Determine a discrete degree value according to each selected first local abnormal factor value, and judge whether the discrete degree value is less than a preset discrete threshold value; If not, for each first local anomaly factor value, determining a difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter; If yes, the method further comprises: Make sure there are no outliers.

4. The method according to claim 3, characterized in that Determining the discrete degree value according to each selected first local abnormal factor value includes: The variance or standard deviation of each selected first local abnormal factor value is determined, and the variance or standard deviation is used as the discrete degree value.

5. The method according to claim 1, characterized in that If the second difference is less than the outlier detection parameter value, determining the local anomaly factor value before the third local anomaly factor value as the target local anomaly factor value comprises: If the second difference is smaller than the outlier detection parameter value, the second difference values ​​corresponding to the fourth number of third local anomaly factor values ​​are successively preset after the third local anomaly factor value, and if each of the obtained second differences is smaller than the outlier detection parameter value, the local anomaly factor value before the third local anomaly factor value is determined as the target local anomaly factor value.

6. An outlier detection device, characterized in that: The device comprises: A first determination module is used to form a data set from each acquired log in the process of detecting password brute force attack behavior, perform feature extraction on each log, obtain a feature vector corresponding to each log, and the feature vector corresponding to each log is each data point in the data set; determine the local anomaly factor value of each data point in the data set, and select a preset first number of first local anomaly factor values ​​in descending order of the local anomaly factor values; A second determination module is used to determine, for each first local anomaly factor value, a first difference between the first local anomaly factor value and a preset second number of second local anomaly factor values ​​thereafter; and determine an outlier detection parameter value according to each determined first difference; A third determination module is used to determine a target local anomaly factor value according to the outlier detection parameter value, and determine the outliers corresponding to the target local anomaly factor value; The third determination module is specifically used to determine, for each third local anomaly factor value after the first local anomaly factor value, a second difference between the third local anomaly factor value and a preset third number of fourth local anomaly factor values ​​thereafter; if the second difference is less than the outlier detection parameter value, determine the local anomaly factor value before the third local anomaly factor value as the target local anomaly factor value.

7. The device according to claim 6, characterized in that The second determination module is specifically configured to use the minimum value of each determined first difference as the outlier detection parameter value.

8. The device according to claim 6, characterized in that The device further includes: a fourth determining module and a fifth determining module; The fourth determination module is used to determine a discrete degree value according to each selected first local abnormal factor value, and judge whether the discrete degree value is less than a preset discrete threshold value; if not, trigger the second determination module, and if yes, trigger the fifth determination module; The fifth determination module is used to determine whether there is no outlier.

9. The device according to claim 8, characterized in that The fourth determination module is specifically used to determine the variance or standard deviation of each selected first local abnormal factor value, and use the variance or standard deviation as the discrete degree value.

10. The device according to claim 6, characterized in that The third determination module is specifically used to determine, if the second difference is smaller than the outlier detection parameter value, the second difference values ​​corresponding to the third local anomaly factor values ​​respectively preset in succession after the third local anomaly factor value, and if each of the obtained second differences is smaller than the outlier detection parameter value, determine the local anomaly factor value before the third local anomaly factor value as the target local anomaly factor value.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 5 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 5 are implemented.

Images