Abnormal data detection method, device, electronic device and storage medium

Abstract

The present invention provides an abnormal data detection method, apparatus, electronic device, and storage medium, relating to the field of network security technology. The method comprises calculating a probability distribution of normal domain name data; calculating a first feature value corresponding to multiple features to be tested for the domain name to be tested; calculating a score for the first feature value based on the probability distribution; and determining whether the domain name to be tested is abnormal data based on the score and a preset abnormality threshold. Using the domain name string itself for detection, the method enables fast and efficient domain name anomaly detection, resolving the issues with existing methods, such as the inability to detect in real time or the need to obtain additional information for detection.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and more specifically, to an abnormal data detection method, device, electronic device, and storage medium. Background Art

[0002] The DNS protocol is an essential network communication protocol on the Internet and in enterprise intranets. Network devices and perimeter defense equipment rarely filter, analyze, or block DNS traffic. Therefore, hiding data or instructions within the DNS protocol for transmission is a covert and effective method. When an attacker compromises a server or a server is infected with malware, worms, or Trojans, transmitting abnormal data through DNS domain names is a simple and effective method widely exploited by attackers in attack scenarios that utilize DNS tunneling.

[0003] Abnormal domain names differ significantly from normal domain names. For example, they are typically longer and often contain non-human-readable strings with chaotic character distribution. Therefore, detecting DNS domain names can effectively identify DNS anomalies such as DNS tunneling. Existing detection methods sometimes require obtaining multiple domain name data points, while others require obtaining additional information about the domain name's internet characteristics. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for detecting abnormal data, which uses the domain name string itself for detection, and can quickly and efficiently detect anomalies in domain names, thereby solving the problem that existing methods cannot detect in real time or require additional information to perform detection.

[0005] The present invention provides a method for detecting abnormal data, the method comprising:

[0006] Calculate the probability distribution of normal domain name data;

[0007] Calculating first feature values ​​corresponding to multiple features to be tested of the domain name to be tested;

[0008] Calculating a score of the first eigenvalue based on the probability distribution;

[0009] Determine whether the domain name to be tested is abnormal data based on the score and a preset abnormality threshold.

[0010] In the above implementation process, the present application performs DNS domain name anomaly detection only from the domain name string itself, without resorting to any information other than the domain name string, which is fast, efficient and easy to implement; a large number of normal events occur on both sides of the peak of the probability distribution, while abnormal events usually deviate from the behavioral baseline of normal events and are located at the edge of the probability distribution. Therefore, the probability distribution of normal domain name data can be used to detect abnormal domain names with high accuracy, which solves the problem that existing methods cannot detect in real time or require additional information to perform detection.

[0011] Furthermore, the calculation of the probability distribution of normal domain name data includes:

[0012] Obtain normal domain name data for various application scenarios;

[0013] Extracting normal features of the normal domain name data;

[0014] Calculating a second feature value corresponding to each normal feature of the normal domain name data;

[0015] A probability distribution is calculated based on the second eigenvalue.

[0016] In the above implementation process, normal domain name data in various application scenarios is obtained, thereby covering multiple application scenarios with a high probability, and a probability distribution can be obtained based on the probability that the second eigenvalue of the normal feature falls within the value range.

[0017] Furthermore, the calculating the probability distribution based on the second eigenvalue includes:

[0018] Obtain a value range of a second characteristic value of any first normal characteristic;

[0019] Divide the value range into k value intervals, where k is a set integer value;

[0020] Initialize the probability value of each value interval to 1 / k;

[0021] Obtaining a second characteristic value of a second normal characteristic;

[0022] Determine a value interval of the second eigenvalue and update the probability value corresponding to the value interval by adding one to both the numerator and denominator. Correspondingly, update the probability values ​​of other value intervals by adding one to the denominator.

[0023] The probability value is updated using all second feature values ​​of each normal feature to obtain a probability distribution of each normal feature.

[0024] In the above implementation process, the probability distribution of normal features is calculated so that the score of the domain name to be tested can be obtained according to the probability distribution.

[0025] Furthermore, calculating the score of the first eigenvalue based on the probability distribution includes:

[0026] Obtaining a value interval of the first eigenvalue in the probability distribution;

[0027] Determining a probability value in the probability distribution based on the value interval;

[0028] Add one to both the numerator and denominator of the probability value to obtain a base probability value;

[0029] Add one to the denominator of the probability value corresponding to other value intervals in the probability distribution to obtain a new probability value;

[0030] Accumulate all probability values ​​that are less than or equal to the benchmark probability value to obtain the probability sum;

[0031] Based on the probability and calculation of the score corresponding to the domain name to be tested:

[0032] A=-log 10 (p-sum);

[0033] Among them, p-sum represents the probability sum.

[0034] In the above implementation process, for the domain name to be tested, the probability value of the first eigenvalue of the domain name to be tested is obtained in the probability distribution, and a score is calculated based on the probability value. The score is used as the basis for judging whether the domain name to be tested is abnormal, which can quickly and efficiently determine the abnormality of the domain name to be tested.

[0035] Furthermore, the preset abnormality threshold includes a first abnormality score threshold and an abnormality score number threshold, and determining whether the domain name to be tested is abnormal data based on the score and the preset abnormality threshold includes:

[0036] Comparing the multiple scores of the domain name to be tested with the first anomaly score threshold, and recording the number of features to be tested whose scores are not less than the first anomaly score threshold;

[0037] If the number is not less than the abnormal score number threshold, the domain name to be tested is abnormal data.

[0038] In the above implementation process, judging from the size and number of the scores of the domain names to be tested can obtain a relatively accurate judgment result.

[0039] Furthermore, the calculating the probability distribution based on the second eigenvalue includes:

[0040] Set the number of value intervals corresponding to each second eigenvalue to be k1, k2, ..., k N , then the total number of value intervals of the normal feature is k1·k2·…·k N, where N represents the number of normal features;

[0041] Initialize the probability value of each value interval to 1 / (k1·k2·…·k N );

[0042] Traversing all second eigenvalues ​​to determine a value range of the second eigenvalue;

[0043] The probability value corresponding to the value interval is updated by adding one to both the numerator and the denominator. Correspondingly, the probability values ​​of other value intervals are updated by adding one to the denominator to obtain the probability distribution.

[0044] In the above implementation process, as another approach, all features of all normal domain name data may be put together to establish a joint feature, thereby constructing a probability distribution of the joint feature.

[0045] Furthermore, the preset abnormality threshold includes a second abnormality score threshold, and determining whether the domain name to be tested is abnormal data based on the score and the preset abnormality threshold includes:

[0046] Comparing the score of the domain name to be tested with the second anomaly score threshold;

[0047] If the score is not less than the second abnormal score threshold, the domain name to be tested is abnormal data.

[0048] In the above implementation process, for the probability distribution of the joint features, the domain name to be tested may be compared with the second anomaly score threshold to determine whether it is abnormal data.

[0049] Before the step of calculating first feature values ​​corresponding to the plurality of features to be tested of the domain name to be tested, the method further includes:

[0050] Using a delimiter to split the string of the domain name to be tested to obtain multiple substrings;

[0051] Anomaly detection is performed on the substring.

[0052] The above implementation process is helpful in detecting some character strings that appear abnormal, with high accuracy.

[0053] The present invention also provides an abnormal data detection device, comprising:

[0054] Probability distribution calculation module, used to calculate the probability distribution of normal domain name data;

[0055] A feature value calculation module to be tested, used to calculate a first feature value corresponding to a plurality of features to be tested of the domain name to be tested;

[0056] A score acquisition module, configured to calculate a score of the first eigenvalue based on the probability distribution;

[0057] The anomaly detection module is used to determine whether the domain name to be tested is abnormal data based on the score and a preset anomaly threshold.

[0058] In the above implementation process, the present application performs DNS domain name anomaly detection only from the domain name string itself, without resorting to any information other than the domain name string, which is fast, efficient and easy to implement; a large number of normal events occur on both sides of the peak of the probability distribution, while abnormal events usually deviate from the behavioral baseline of normal events and are located at the edge of the probability distribution. Therefore, the probability distribution of normal domain name data can be used to detect abnormal domain names with high accuracy, which solves the problem that existing methods cannot detect in real time or require additional information to perform detection.

[0059] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform any of the above-mentioned abnormal data detection methods.

[0060] An embodiment of the present application further provides a readable storage medium, wherein the readable storage medium stores computer program instructions. When the computer program instructions are read and executed by a processor, any of the above-mentioned abnormal data detection methods is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 A flowchart of an abnormal data detection method provided in an embodiment of the present application;

[0063] Figure 2 Flowchart of the DNS domain name anomaly detection method provided in an embodiment of the present application;

[0064] Figure 3 A flowchart for calculating the probability distribution of normal domain name data provided in an embodiment of the present application;

[0065] Figure 4 A flowchart for calculating the probability distribution of each feature provided in the embodiment of the present application;

[0066] Figure 5A flowchart for calculating the score of a domain name to be tested provided in an embodiment of the present application;

[0067] Figure 6 This is a flowchart of abnormal data judgment provided by the embodiment of the present application;

[0068] Figure 7 A flowchart of the probability distribution calculation of the joint features provided in the embodiment of the present application;

[0069] Figure 8 Another abnormal data judgment flow chart provided in an embodiment of the present application;

[0070] Figure 9 A structural block diagram of an abnormal data detection device provided in an embodiment of the present application;

[0071] Figure 10 This is a structural block diagram of another abnormal data detection device provided in an embodiment of the present application.

[0072] icon:

[0073] 100-Probability distribution calculation module; 101-Probability distribution module for each feature; 102-Joint feature probability distribution module; 200-Test feature value calculation module; 300-Score acquisition module; 400-Anomaly detection module; 401-Multi-threshold judgment module; 402-Single threshold judgment module. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0075] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0076] Example 1

[0077] Please see Figure 1 , Figure 1 This is a flowchart of an abnormal data detection method provided in an embodiment of the present application. This method detects DNS domain name anomalies only from the domain name string itself, which is fast, efficient, and easy to implement.

[0078] The specific principle is this: among a large number of events, abnormal events are always among the minority. The probability of an event forms a probability distribution. Normal events, such as normal DNS domain name data, occur in large numbers on both sides of the probability distribution's peak. However, abnormal events, such as abnormal DNS domain name data, typically deviate from the baseline of normal behavior. Therefore, the probability of an abnormal event is always located at the edge of the probability distribution. In other words, compared to events near the peak of the probability distribution, those at the edge are more likely to be abnormal events. Therefore, the probability distribution of normal events can be used to judge the event under test.

[0079] This application combines this theory with the DNS domain name anomaly detection scenario and proposes a DNS domain name anomaly detection method. The solution first needs to collect a large amount of normal DNS domain name data, then use various features that reflect the characteristics of abnormal DNS domain names to calculate the probability distribution, then calculate the anomaly score of the domain name to be tested, and finally use the anomaly score to judge whether the unknown DNS domain name is abnormal. Figure 2 FIG. 1 is a flowchart of a method for detecting anomalies in a DNS domain name, which specifically includes the following steps:

[0080] Step S100: Calculate the probability distribution of normal domain name data;

[0081] Step S200: Calculating first feature values ​​corresponding to multiple features to be tested of the domain name to be tested;

[0082] Step S300: Calculating the score of the first eigenvalue based on the probability distribution;

[0083] Step S400: determining whether the domain name to be tested is abnormal data based on the score and a preset abnormality threshold.

[0084] This application performs DNS domain name anomaly detection only based on the domain name string itself, without the need for any information other than the domain name string. It is fast, efficient, and easy to implement. A large number of normal events occur on both sides of the peak of the probability distribution, while abnormal events usually deviate from the behavioral baseline of normal events and are located at the edge of the probability distribution. Therefore, the probability distribution of normal domain name data can be used to detect abnormal domain names with high accuracy, solving the problem that existing methods cannot detect in real time or require obtaining additional information for detection.

[0085] Among them, such as Figure 3 FIG. 1 is a flow chart for calculating the probability distribution of normal domain name data. Step S100 specifically includes:

[0086] Step S110: Acquire normal domain name data from various application scenarios;

[0087] Collect normal domain name data - Since the meaning of abnormal domain names may be different in different application scenarios, in different application scenarios, it is only necessary to collect normal domain names in the corresponding application scenarios in a targeted manner, so as to cover as many application scenarios as possible and improve the comprehensiveness of normal domain name data.

[0088] Step S120: extracting normal features of the normal domain name data;

[0089] Abnormal DNS domain names usually have the following characteristics: longer length, more chaotic character distribution, and the character composition of abnormal domain names is significantly different from that of normal domain names.

[0090] Based on these characteristics, many features can be extracted to reflect the characteristics of abnormal domain names. The optional features include: the length of the domain name, the number of numeric characters in the domain name, the proportion of non-lowercase characters in the domain name, the number of different characters in the domain name, the proportion of vowels in the domain name, the proportion of consonants in the domain name, the ratio of vowels to consonants in the domain name, the ratio of the sum of the lengths of consecutive consonant strings in the domain name to the domain name length, the maximum length of consecutive consonant strings in the domain name, the ratio of the sum of the lengths of consecutive numeric strings in the domain name to the domain name length, the maximum length of consecutive numeric strings in the domain name, the number of labels in the domain name (the string after the domain name is separated by '.' is called a label), the maximum length of labels in the domain name, the average length of labels in the domain name, the maximum length of tokens in the domain name (the string after the domain name is separated by special symbols is called a token), the average length of tokens in the domain name, the number of numeric-letter conversions in the domain name, the number of uppercase and lowercase letter conversions in the domain name, the Gini coefficient of the domain name, the 2-gram entropy of the domain name, the 3-gram entropy of the domain name, and the character transition probability of the domain name.

[0091] Therefore, the above features can be extracted from normal domain name data to increase the difference between normal domain name data and abnormal domain name data, thereby improving the accuracy of the detection results.

[0092] Step S130: Calculating a second feature value corresponding to each normal feature of the normal domain name data;

[0093] Step S140: Calculate probability distribution based on the second eigenvalue.

[0094] Calculate the eigenvalues ​​(second eigenvalues) of each normal feature corresponding to the normal domain name data, determine the value range of each normal feature, divide the value range into reasonable value intervals, calculate the probability of each value interval, and obtain the probability distribution corresponding to the normal feature.

[0095] As one of the implementation methods, Figure 4FIG. 1 is a flow chart for calculating the probability distribution of each feature. Step S140 specifically includes the following steps:

[0096] Step S141: Obtaining a value range of a second characteristic value of any first normal characteristic;

[0097] Step S142: Divide the value range into k value intervals, where k is a set integer value;

[0098] Step S143: Initialize the probability value of each value interval to 1 / k;

[0099] Step S144: obtaining a second characteristic value of the second normal characteristic;

[0100] Step S145: Determine the value interval of the second eigenvalue and update the probability value corresponding to the value interval to the numerator and denominator plus one, that is, (1+1) / (k+1). Correspondingly, the probability values ​​of other value intervals are updated to the denominator plus one, that is, 1 / (k+1);

[0101] Step S146: using all second feature values ​​of each normal feature to update the probability value to obtain a probability distribution of each normal feature.

[0102] It should be noted that the first normal feature and the second normal feature mentioned above refer to the same feature, such as domain name length.

[0103] Taking a normal feature of a normal domain name as an example, assuming that the value range of the feature value of the normal feature is [a, b], the value range of the feature is divided into k value intervals, and for each interval, the initialization probability value is set to 1 / k.

[0104] Take each normal domain name one by one, calculate the eigenvalue of the feature, record the value range in which the eigenvalue falls, and then update the probability value corresponding to the range. The updating method is to add 1 to both the numerator and denominator, that is, (1+1) / (k+1); then, update the probability value of other value ranges to 1 / (k+1), that is, add 1 to the denominator; at this time, the probability value corresponding to each value range is updated.

[0105] Repeat the above steps. When all normal domain name data are obtained, the probability corresponding to each value interval is obtained. These probability values ​​constitute the probability distribution of each normal feature.

[0106] For the domain name to be tested, take one feature as an example and calculate the feature value corresponding to the domain name to be tested, and then calculate the anomaly score corresponding to this feature value. For other features, use the same method to calculate the anomaly score corresponding to each feature. Assuming N features are selected, the domain name to be tested will have N anomaly scores.

[0107] like Figure 5FIG. 3 is a flowchart of calculating the score of a domain name to be tested. Step S300 may specifically include the following steps:

[0108] Step S301: Obtain the value interval of the first eigenvalue in the probability distribution;

[0109] Step S302: determining a probability value in the probability distribution based on the value interval;

[0110] Step S303: adding one to both the numerator and the denominator of the probability value to obtain a reference probability value;

[0111] Step S304: adding one to the denominator of the probability value corresponding to other value intervals in the probability distribution to obtain a new probability value;

[0112] Step S305: Accumulate all probability values ​​that are less than or equal to the reference probability value to obtain a probability sum;

[0113] Step S306: Calculate the score corresponding to the domain name to be tested based on the probability sum:

[0114] A=-log 10 (p-sum);

[0115] Among them, p-sum represents the probability sum.

[0116] For the domain name to be tested, calculate the eigenvalue corresponding to a certain feature, record the value range of the eigenvalue, search the probability corresponding to the value range in the probability distribution corresponding to the feature, add 1 to both the numerator and denominator of the probability to obtain the baseline probability value, recorded as p; then, add 1 to the denominator of the probability of other value ranges to obtain the new probability value. Compare the new probability values ​​corresponding to all value ranges with p, take out all new probability values ​​less than or equal to p and accumulate them, and record the accumulated sum as p-sum. The anomaly score A of the domain name to be tested is -log 10 (p-sum).

[0117] like Figure 6 FIG. 4 is a flowchart of abnormal data determination, wherein step S400 may specifically include the following steps:

[0118] Step S411: comparing the multiple scores of the domain name to be tested with the first anomaly score threshold, and recording the number of features to be tested whose scores are not less than the first anomaly score threshold;

[0119] Step S412: If the number is not less than the abnormal score number threshold, the domain name to be tested is abnormal data.

[0120] Set a first anomaly score threshold α and an anomaly score number threshold β; compare the N scores corresponding to the domain name to be tested with the threshold α respectively, and record the number of anomaly scores count greater than or equal to α; compare count with the threshold β. If count is greater than or equal to β, the domain name to be tested is considered abnormal; if count is less than β, the domain name to be tested is considered normal.

[0121] For example, in a certain application scenario, 1,000 normal domain names are collected. The process of using these 1,000 normal domain name data to detect the domain name data to be tested is as follows:

[0122] If the number of normal features selected is 3, the three normal features selected are: domain name length, domain name 2-gram entropy, and domain name 3-gram entropy. According to the actual situation, the value range of domain name length is set to [0,50], (50,100], (100,150], (150,200], (200,250], and [other]. The value range of domain name 2-gram entropy is set to [0,0.2], (0.2,0.4], (0.4,0.6], (0.6,0.8], and (0.8,1.0). The value range of domain name 3-gram entropy is set to [0,0.2], (0.2,0.4], (0.4,0.6], (0.6,0.8], and (0.8,1.0].

[0123] 1000 normal domain name data are used to construct the probability distribution corresponding to the three features.

[0124] There is a domain name to be tested, whose length is 170, 2-gram entropy is 0.53, and 3-gram entropy is 0.27.

[0125] Calculate the anomaly score using domain name length: The domain name length interval is 6, and the initial probability distribution is [1 / 6, 1 / 6, 1 / 6, 1 / 6, 1 / 6, 1 / 6]. Combining the above process with 1000 normal domain name data, we calculate the domain name length probability distribution, resulting in a probability distribution of [907 / 1006, 83 / 1006, 12 / 1006, 2 / 1006, 1 / 1006, 1 / 1006]. Since the domain name under test is 170 characters long, it falls in the fourth interval. Therefore, the probability of this interval is updated to 3 / 1007, and the probability distribution is updated to [907 / 1007, 83 / 1007, 12 / 1007, 3 / 1007, 1 / 1007, 1 / 1007]. The probability distribution is searched for probability values ​​less than or equal to 3 / 1007, which are 3 / 1007, 1 / 1007, and 1 / 1007, respectively. Then p-sum = 3 / 100 + 1 / 1007 + 1 / 1007 = 5 / 1007, and the anomaly score A1 = -log10(17 / 1007) = 2.304.

[0126] Calculate the anomaly score using the domain name's 2-gram entropy: The domain name's 2-gram entropy interval is 5, and the initial probability distribution is [1 / 5, 1 / 5, 1 / 5, 1 / 5, 1 / 5]. Using the above process, we calculate the probability distribution of the domain name's 2-gram entropy based on 1000 normal domain names, resulting in a probability distribution of [832 / 1005, 170 / 1005, 1 / 1005, 1 / 1005, 1 / 1005]. Since the domain name under test has a 2-gram entropy of 0.53, it falls in the third interval. Therefore, the probability of this interval is updated to 2 / 1006, and the probability distribution is updated to [832 / 1006, 170 / 1006, 2 / 1006, 1 / 1006, 1 / 1006]. We search the probability distribution for probability values ​​less than or equal to 2 / 1006, which are 2 / 1006, 1 / 1006, and 1 / 1006, respectively. Then p-sum = 2 / 1006 + 1 / 1006 + 1 / 1006 = 4 / 1006, and the anomaly score A2 = -log10(4 / 1006) = 2.400.

[0127] Calculate the anomaly score using the domain name's 3-gram entropy: The domain name's 3-gram entropy interval is 5, and the initial probability scores are [1 / 5, 1 / 5, 1 / 5, 1 / 5, 1 / 5]. Using the above process, we calculate the probability distribution of the domain name's 3-gram entropy based on 1000 normal domain names, resulting in a probability distribution of [974 / 1005, 28 / 1005, 1 / 1005, 1 / 1005, 1 / 1005]. Since the domain name under test has a 3-gram entropy of 0.27, it falls in the second interval. Therefore, the probability of this interval is updated to 29 / 1006, and the probability distribution is updated to [974 / 1006, 29 / 1006, 1 / 1006, 1 / 1006, 1 / 1006].

[0128] Searching for probability values ​​less than or equal to 29 / 1006 in the probability distribution yields 29 / 1006, 1 / 1006, 1 / 1006, and 1 / 1006, respectively. Thus, p-sum = 29 / 1006 + 1 / 1006 + 1 / 1006 + 1 / 1006 = 32 / 1006, and the anomaly score A3 = -log10(32 / 1006) = 1.497.

[0129] Set the anomaly score threshold α to 1.0 and the anomaly score number threshold β to 2. The anomaly scores corresponding to the three features are A1 = 2.304, A2 = 2.400, and A3 = 1.497. The number of anomaly scores greater than or equal to 1.0 (threshold α) is 3, which is greater than the anomaly score threshold of 2. Therefore, the domain name under test is considered an anomaly.

[0130] As another embodiment, Figure 7 FIG. 1 is a flowchart of calculating the probability distribution of joint features, wherein step S140 specifically includes the following steps:

[0131] Step S147: Set the number of value intervals corresponding to each second eigenvalue to be k1, k2, ..., k N , then the total number of value intervals of the normal feature is k1·k2·…·k N , where N represents the number of normal features;

[0132] For each domain name, there are N value intervals corresponding to the second eigenvalue. These N value intervals constitute the value vector of each domain name, that is, each domain name corresponds to a 1×N dimensional value vector. The number of value intervals of the i-th dimension feature of the 1×N dimensional value vector is k i , that is, the value of the i-th dimension feature is k i There are possibilities, so the values ​​of 1×N are k1·k2·…·k N possibilities.

[0133] Step S148: Initialize the probability value of each value interval to 1 / (k1·k2·…·k N );

[0134] Step S149: traverse all second eigenvalues ​​to determine the value range of the second eigenvalue;

[0135] Step S150: Update the probability value corresponding to the value interval by adding one to both the numerator and the denominator, i.e. (1+1) / (k1·k2·…·k N +1), and accordingly, the probability values ​​of other value intervals are updated to the denominator plus one, that is, 1 / (k1·k2·…·k N +1), and obtain the probability distribution.

[0136] Assume that N normal features are jointly selected from the normal domain name data, and the number of value intervals corresponding to each normal feature is k1, k2, ..., k N , then the total number of value intervals corresponding to N features is k1·k2·…·k N At this time, the probability of each value interval is initialized to 1 / (k1·k2·…·k N The calculation process for the joint feature probability distribution is as follows: select normal domain names one by one, then calculate the N eigenvalues ​​corresponding to the normal domain names, determine the single value interval in which the N eigenvalues ​​fall, and add 1 to the numerator and denominator of the probability corresponding to the value interval; then, add 1 to the denominator of the probability of the other value intervals; after traversing all normal domain name data, the probability of each value interval is obtained, and these probabilities constitute the probability distribution of the joint feature.

[0137] For the domain name to be tested, calculate the eigenvalues ​​of N features, form a eigenvector from the eigenvalues ​​of N features, record the single value interval in which the eigenvector falls, add 1 to both the numerator and denominator of the probability corresponding to the value interval, record this probability value as p, and then add 1 to the denominator of the probability of other value intervals; compare the probability values ​​of all value intervals with p, take out all probability values ​​less than or equal to p and add them up, and record the accumulated sum as p-sum. Then the anomaly score A of the domain name to be tested under the joint feature probability distribution is -log 10 (p-sum).

[0138] like Figure 8 FIG. 4 is another abnormal data determination flow chart, wherein step S400 may specifically include the following steps:

[0139] Step S421: Compare the score of the domain name to be tested with the second anomaly score threshold;

[0140] Step S422: If the score is not less than the second abnormal score threshold, the domain name to be tested is abnormal data.

[0141] The second anomaly score threshold is set to γ. If the anomaly score of the domain name to be tested is greater than or equal to γ, the domain name to be tested is considered to be abnormal.

[0142] It should be noted that the domain name to be tested can be a complete domain name string or a segmented substring: the domain name string to be tested is segmented using a delimiter to obtain multiple substrings, and abnormal data detection is performed on the substrings.

[0143] In the above calculation process, the domain name itself is the target of anomaly detection. Therefore, analysis of actual domain name usage reveals that, in most cases, an abnormal domain name is not the entire domain name string, but rather a portion of it. Therefore, abnormal domain names can also be detected by substrings of the domain name string separated by the '.' and '-' symbols. Compared to detecting the entire domain name string, detecting substrings has a higher accuracy rate. This choice can be made based on actual circumstances and is not limited here.

[0144] For example, in the same application environment, the data is still 1000 normal domain name data. The process of using the joint distribution anomaly detection method is as follows:

[0145] We use domain name length, domain name 2-gram entropy, and domain name 3-gram entropy as examples. Similarly, the domain name length range is set to [0, 50], (50, 100], (100, 150], (150, 200], (200, 250], [other]. The domain name 2-gram entropy range is set to [0, 0.2], (0.2, 0.4], (0.4, 0.6], (0.6, 0.8], (0.8, 1.0). The domain name 3-gram entropy range is set to [0, 0.2], (0.2, 0.4], (0.4, 0.6], (0.6, 0.8], (0.8, 1.0].

[0146] The number of value intervals of the domain name length k1=6, the number of value intervals of the domain name 2-gram entropy k2=5, and the number of value intervals of the domain name 3-gram entropy k3=5.

[0147] At this point, the number of intervals in the joint probability distribution is K = k1·k2·k3 = 150. The probability values ​​for each interval in the joint probability distribution are initialized to [1 / 150,…,1 / 150]. Using 1000 normal domain name data to calculate the joint probability distribution, the resulting probability distribution is [832 / 1150, 76 / 1150, 68 / 1150, 16 / 1150, 12 / 1150, 2 / 1150, 1 / 1150 (a total of 144)].

[0148] The length of the domain name to be tested is 56, the 2-gram entropy is 0.61, and the 3-gram entropy is 0.42.

[0149] The eigenvector formed by the eigenvalues ​​of the domain name under test falls in the 23rd interval (2*4*3-1). The probability of this interval is updated to 2 / 1151, and the joint probability distribution is updated to [832 / 1151, 76 / 1151, 68 / 1151, 16 / 1151, 12 / 1151, 2 / 1151 (two total), 1 / 1151 (143 total)]. The joint probability distribution is searched for probabilities less than or equal to 2 / 1151, which are 2 / 1151 (two total) and 1 / 1151 (143 total). Therefore, p-sum = (2 / 1151)*2 + (1 / 1151)*143 = 147 / 1151, and the anomaly score A = -log10(147 / 1151) = 0.894.

[0150] The anomaly score threshold γ is set to 1.0. Since the anomaly score of the domain name to be tested is less than the threshold, the domain name to be tested is determined to be normal.

[0151] Based on normal DNS domain data, the system calculates the probability distribution of each feature individually or selects features to calculate a joint probability distribution. For the domain to be tested, the feature value is calculated, substituted into the corresponding probability distribution, and the anomaly score is calculated. The anomaly domain identification method is then used to determine the domain to be tested. This calculation process is fast and efficient, allowing for rapid anomaly determination of unknown events.

[0152] This method only requires normal domain name data as basic data to build a detection model. After the detection model is built, domain name anomalies are detected only based on the domain name string itself, without the need for any information other than the domain name string. It is simple and efficient.

[0153] Example 2

[0154] The embodiment of the present application provides an abnormal data detection device, which is applied to the abnormal data detection method described in Example 1. Figure 9 FIG. 1 is a block diagram of a device for detecting abnormal data, which includes but is not limited to:

[0155] Probability distribution calculation module 100, used to calculate the probability distribution of normal domain name data;

[0156] The feature value calculation module 200 is used to calculate the first feature value corresponding to the multiple features to be tested of the domain name to be tested;

[0157] A score acquisition module 300, configured to calculate a score of the first eigenvalue based on the probability distribution;

[0158] The anomaly detection module 400 is configured to determine whether the domain name to be tested is abnormal data based on the score and a preset anomaly threshold.

[0159] Among them, such as Figure 10 As shown in FIG. 1 , it is a structural block diagram of another abnormal data detection device. Figure 9 On the basis of, the probability distribution calculation module 100 includes a probability distribution module 101 for each feature and a joint feature probability distribution module 102, wherein the probability distribution module 101 for each feature is used to calculate the probability distribution of each normal feature; the joint feature probability distribution module 102 is used to calculate the probability distribution of the joint feature. The specific calculation process has been specifically described in Example 1 and will not be repeated here.

[0160] The anomaly detection module 400 includes a multi-threshold judgment module 401 and a single-threshold judgment module 402. The multi-threshold judgment module 401 is used to make judgments based on a first anomaly score threshold and an anomaly score number threshold; the single-threshold judgment module 402 is used to make judgments based on a second anomaly score threshold. The specific judgment process has been specifically described in Example 1 and will not be repeated here.

[0161] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the abnormal data detection method described in Example 1.

[0162] An embodiment of the present application further provides a readable storage medium, wherein the readable storage medium stores computer program instructions. When the computer program instructions are read and executed by a processor, the abnormal data detection method described in Example 1 is executed.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0164] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0165] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0166] The foregoing is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.

[0167] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0168] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

Claims

1. A method for detecting abnormal data, characterized in that: The method comprises: Calculate the probability distribution of normal domain name data; Calculating first feature values ​​corresponding to multiple features to be tested of the domain name to be tested; Calculating the score of the first eigenvalue based on the probability distribution specifically includes: obtaining a value interval of the first eigenvalue in the probability distribution; determining a probability value in the probability distribution based on the value interval; adding one to both the numerator and the denominator of the probability value to obtain a baseline probability value; adding one to the denominator of the probability values ​​corresponding to other value intervals in the probability distribution to obtain a new probability value; accumulating all new probability values ​​that are less than or equal to the baseline probability value to obtain a probability sum; and calculating a score corresponding to the domain name to be tested based on the probability sum: A=-log 10 (p-sum); where p-sum represents probability sum; Determine whether the domain name to be tested is abnormal data based on the score and a preset abnormality threshold.

2. The abnormal data detection method according to claim 1, characterized in that: Calculating the probability distribution of normal domain name data includes: Obtain normal domain name data for various application scenarios; Extracting normal features of the normal domain name data; Calculating a second feature value corresponding to each normal feature of the normal domain name data; A probability distribution is calculated based on the second eigenvalue.

3. The abnormal data detection method according to claim 2, characterized in that: The calculating the probability distribution based on the second eigenvalue includes: Obtain a value range of a second characteristic value of any first normal characteristic; Divide the value range into k value intervals, where k is a set integer value; Initialize the probability value of each value interval to 1 / k; Obtaining a second characteristic value of a second normal characteristic; Determine a value interval of the second eigenvalue and update the probability value corresponding to the value interval by adding one to both the numerator and denominator. Correspondingly, update the probability values ​​of other value intervals by adding one to the denominator. The probability value is updated using all second feature values ​​of each normal feature to obtain a probability distribution of each normal feature.

4. The abnormal data detection method according to claim 1, characterized in that: The preset abnormality threshold includes a first abnormality score threshold and an abnormality score number threshold, and determining whether the domain name to be tested is abnormal data based on the score and the preset abnormality threshold includes: Comparing the multiple scores of the domain name to be tested with the first anomaly score threshold, and recording the number of features to be tested whose scores are not less than the first anomaly score threshold; If the number is not less than the abnormal score number threshold, the domain name to be tested is abnormal data.

5. The abnormal data detection method according to claim 2, characterized in that: The calculating the probability distribution based on the second eigenvalue includes: Set the number of value intervals corresponding to each second eigenvalue to be k1, k2, ..., k N , then the total number of value intervals of the normal feature is k1·k2·…·k N , where N represents the number of normal features; Initialize the probability value of each value interval to 1 / (k1·k2·…·k N ); Traversing all second eigenvalues ​​to determine a value range of the second eigenvalue; The probability value corresponding to the value interval is updated by adding one to both the numerator and the denominator. Correspondingly, the probability values ​​of other value intervals are updated by adding one to the denominator to obtain the probability distribution.

6. The abnormal data detection method according to claim 5, characterized in that: The preset abnormality threshold includes a second abnormality score threshold, and determining whether the domain name to be tested is abnormal data based on the score and the preset abnormality threshold includes: Comparing the score of the domain name to be tested with the second anomaly score threshold; If the score is not less than the second abnormal score threshold, the domain name to be tested is abnormal data.

7. The abnormal data detection method according to claim 1, characterized in that: Before the step of calculating first feature values ​​corresponding to the plurality of features to be tested of the domain name to be tested, the method further includes: Using a delimiter to split the string of the domain name to be tested to obtain multiple substrings; Anomaly detection is performed on the substring.

8. An abnormal data detection device, characterized in that: The device comprises: Probability distribution calculation module, used to calculate the probability distribution of normal domain name data; A feature value calculation module to be tested, used to calculate a first feature value corresponding to a plurality of features to be tested of the domain name to be tested; The score acquisition module is used to calculate the score of the first eigenvalue based on the probability distribution, specifically: obtaining the value range of the first eigenvalue in the probability distribution; determining the probability value in the probability distribution based on the value range; adding one to the numerator and denominator of the probability value to obtain a baseline probability value; adding one to the denominator of the probability values ​​corresponding to other value ranges in the probability distribution to obtain a new probability value; accumulating all new probability values ​​that are less than or equal to the baseline probability value to obtain a probability sum; and calculating the score corresponding to the domain name to be tested based on the probability sum: A=-log 10 (p-sum); where p-sum represents probability sum; The anomaly detection module is used to determine whether the domain name to be tested is abnormal data based on the score and a preset anomaly threshold.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the abnormal data detection method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the abnormal data detection method according to any one of claims 1 to 7 is executed.

Images