Accompanying Domain Name Detection Method Based on Locality Sensitive Hash Algorithm

Abstract

This application provides a method for detecting accompanying domain names based on the Locality Sensitive Hashing (LSH) algorithm, which includes the following steps: calculating spatiotemporal accompanying pairs using LSH, and filtering out intersecting spatiotemporal accompanying pairs; calculating and sorting the true C-Rank scores of the intersecting spatiotemporal accompanying pairs; and grouping the target domain names, taking the accompanying pairs of target domain names with a similarity greater than a similarity threshold as the final spatiotemporal accompanying pairs. In industrial-scale massive data scenarios, this application, based on domain name request accompanying relationships, can quickly calculate domain name accompanying pairs using LSH in linear time complexity.

Description

Technical Field

[0001] This application belongs to the field of information security, specifically relating to a method for detecting accompanying domain names based on the Locality Sensitive Hash algorithm. Background Technology

[0002] DNS (Domain Name System) is one of the core foundational services of the internet today, providing a unified mapping service for domain name address space. As a distributed database that maps domain names to IP addresses, DNS makes internet access more convenient. In recent years, with the booming development of the digital economy, on the one hand, an increasing number of illegal cyber activities have begun to abuse DNS for malicious purposes, such as botnet attacks and domain spoofing phishing attacks. On the other hand, a large amount of malicious domain name detection and research has emerged, primarily aimed at preventing and warning users from accessing these malicious domains, thereby mitigating the threats and damage caused by cyberattacks.

[0003] Among various methods for detecting malicious domains, one category is based on DNS traffic resolution data, utilizing domain request co-occurrence relationships for detection. This is because co-occurring domains typically exhibit homomorphism; that is, benign domains usually co-occur with benign domains, and malicious domains usually co-occur with malicious domains. For example, in 2010, Kazumichi Sato et al. established co-occurrence relationships between two domains using Jaccard distance based on the number of hosts, thereby calculating the co-occurrence score of malicious domains. However, this method does not consider temporal correlation, relying solely on the number of different host IPs accessing the domain and the black-and-white information of the IPs, thus introducing some error. Furthermore, this method requires calculating pairwise co-occurrences, resulting in a computational complexity of O(N^2). 2This method is not suitable for running on massive datasets with billions of records. In 2014, Gao et al. used some known malicious domains as seeds to statistically analyze the relationship between domains and seeds, and used TF-IDF and Xmeans clustering algorithms to extract domain groups. However, this method does not consider the time interval between domain requests, thus grouping domains triggered by different network activities into one group, resulting in severe noise interference. This method also has practicality issues with large-scale data because it requires selecting malicious seed domains as input, and each input involves a clustering analysis, which is very time-consuming in engineering implementation. In 2019, Peng Chengwei et al. proposed a malicious domain detection method based on spatiotemporal association. This method first extracts spatiotemporal association relationships to form an association corpus; then uses the word2vec approach to learn the feature vector of each domain; and finally uses a supervised classification model for classification. However, this method involves many complex stages and requires ideal labeled data because the distribution of black and white samples in real-world data is very uneven, making label acquisition difficult and difficult to apply to large-scale datasets such as billions of DNS traffic records. Summary of the Invention

[0004] To at least partially overcome the problems existing in related technologies, this application provides a method for detecting accompanying domain names based on the locality-sensitive hashing algorithm.

[0005] According to an embodiment of this application, this application provides a method for detecting accompanying domain names based on the locality-sensitive hashing algorithm, characterized by comprising the following steps:

[0006] The locality-sensitive hashing algorithm is used to calculate spatiotemporal adjoint pairs and filter out the intersecting spatiotemporal adjoint pairs.

[0007] Calculate and sort the true C-Rank scores of the spatiotemporal adjoint pairs in the intersection;

[0008] By grouping target domains, the companion pairs of target domains with a similarity greater than the similarity threshold are taken as the final time companion pairs.

[0009] In the above-mentioned method for detecting accompanying domain names based on the Locality Sensitive Hashing (LSH) algorithm, the specific process of calculating spatiotemporal accompanying pairs using the LSH algorithm and filtering out intersectional spatiotemporal accompanying pairs is as follows:

[0010] Coarse screening is performed based on time pairs (FQDNs), and time-paired pairs are calculated.

[0011] Based on the FQDN after initial screening using time association and the corresponding IP tuple (IP, time), a fine screening is performed to calculate spatial association pairs;

[0012] By associating the obtained temporal and spatial pairs, the final intersection of the temporal and spatial pairs is obtained.

[0013] Furthermore, the specific process of coarse screening based on time tuples (FQDN) and calculating time adjoining pairs is as follows:

[0014] Calculate the time vector;

[0015] Calculate the minimum hash fingerprint vector using the time vector;

[0016] Based on the minimum hash fingerprint vector, time-adjoint pairs are calculated using the locality-sensitive hashing algorithm.

[0017] Furthermore, the specific process for calculating the time vector is as follows:

[0018] Convert millisecond-level timestamps (time, FQDN) into second-level time indices based on time tuples (time, FQDN);

[0019] Extract the second-level time index corresponding to the time a domain name appears in a day, assign the value 1 to the corresponding index position, and build the time vector of the domain name.

[0020] Furthermore, the specific process of calculating the minimum hash fingerprint vector using the time vector is as follows:

[0021] Calculate the Jaccard distance between any two time vectors; where the Jaccard distance is expressed as a bitwise AND operation or a bitwise OR operation.

[0022] The Jaccard distance is approximated using the minimum hash function MinHash to obtain a low-dimensional approximate hash fingerprint for each high-dimensional vector.

[0023] Furthermore, the specific process of calculating the time-coupling pair using the locality-sensitive hashing algorithm based on the minimum hash fingerprint vector is as follows:

[0024] Using the locality-sensitive hashing algorithm to divide the dimension L The low-dimensional dense fingerprint vector is divided into b bands, each band having r rows, where... ;

[0025] Assuming two dense fingerprint vectors and If the similarity between the two Jaccard distances is s, then the expected similarity threshold is obtained according to the Locality Sensitive Hashing algorithm. t for ;

[0026] Similarity s Greater than the similarity threshold tThe time-coupling pair is the final time-coupling pair.

[0027] Furthermore, the specific process of calculating spatial adjoint pairs based on the FQDN after initial screening using time association and the corresponding IP tuple (IP, time) is as follows:

[0028] Calculate spatial vectors;

[0029] Calculate the minimum hash fingerprint vector based on the spatial vector;

[0030] Spatial adjoint pairs are calculated using the locality-sensitive hashing algorithm based on the minimum hash fingerprint vector.

[0031] Furthermore, the specific process for calculating the spatial vector is as follows:

[0032] The number of IPs is used as the IP index based on spatial tuples (IP, FQDN);

[0033] Extract the IP indexes that appear in a day, assign a value of 1 to the corresponding index position, and create a spatial vector for that domain name.

[0034] Furthermore, the specific process of calculating the minimum hash fingerprint vector based on the spatial vector is as follows:

[0035] Calculate the Jaccard distance between any two spatial vectors, where the Jaccard distance is expressed as a bitwise AND operation or a bitwise OR operation;

[0036] The Jaccard distance is approximated using the minimum hash function MinHash to obtain a low-dimensional approximate hash fingerprint for each high-dimensional vector.

[0037] Furthermore, the specific process of calculating the spatial adjoint pair using the locality-sensitive hashing algorithm based on the minimum hash fingerprint vector is as follows:

[0038] Using the locality-sensitive hashing algorithm to divide the dimension L The low-dimensional dense fingerprint vector is divided into b bands, each band having r rows, where... ;

[0039] Assuming two dense fingerprint vectors and If the similarity between the two Jaccard distances is s, then the expected similarity threshold is obtained according to the Locality Sensitive Hashing algorithm. t for ;

[0040] Similarity s Greater than the similarity threshold t The temporal companion pair serves as the final spatial companion pair.

[0041] Furthermore, the specific process of calculating and sorting the true C-Rank scores in the intersection spatiotemporal adjoint pairs is as follows:

[0042] Assuming that any intersection of spacetime adjoint pairs is known<fqdn1,fqdn2,s> Then we can calculate:

[0043] The number of times the domain name fqdn1 appears in the data within one day is 'a'.

[0044] The domain name fqdn2 appears b times in the data for one day;

[0045] Within a certain time window, given that domain name fqdn1 appears, the number of times domain name fqdn2 appears is c.

[0046] Within a certain time window, if domain name fqdn2 appears, the number of times domain name fqdn1 appears is d;

[0047] Within a certain time window, the number of IPs that simultaneously access the intersection of domain names fqdn1 and fqdn2 is g;

[0048] According to Bayes' theorem, given that domain name fqdn1 appears, the probability of domain name fqdn2 appearing is... Given that domain name fqdn2 appears, the probability of domain name fqdn1 appearing is: ;

[0049] Using the Kulc coefficient in statistics, the similarity between two intersecting spatiotemporal adjoint pairs can be expressed as:

[0050] ,

[0051] when x and y When they are equal, the function The value is the largest;

[0052] The sigmoid function is introduced to smooth the adjoint intersection counts c and d, as well as the number of adjoint intersection IPs g. Mathematically, it is expressed as:

[0053] ,

[0054] In the formula, Indicates the smoothing coefficient;

[0055] Finally, the C-Rank score of the intersection spacetime adjoint pair is:

[0056] .

[0057] As can be seen from the above specific embodiments of this application, it has at least the following beneficial effects: The accompanying domain name detection method based on the locality-sensitive hash algorithm provided in this application can quickly calculate the accompanying domain name pairs in linear time complexity based on the domain name request companion relationship in industrial-grade massive data scenarios (billions of DNS recursive resolution data).

[0058] It should be understood that the above general description and the following specific embodiments are merely exemplary and illustrative, and do not limit the scope of the claims made in this application. Attached Figure Description

[0059] The accompanying drawings, which are part of the specification of this application, illustrate embodiments of the present application and are used together with the description of the specification to illustrate the principles of the present application.

[0060] Figure 1 A flowchart illustrating a method for detecting accompanying domain names based on the Locality Sensitive Hash algorithm, provided for a specific implementation of this application.

[0061] Figure 2 This is a schematic diagram of a locality-sensitive hashing (LSH) method for detecting accompanying domain names based on the LSH algorithm, provided for a specific implementation of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the spirit of the content disclosed in this application will be clearly explained below with reference to the accompanying drawings and detailed description. After understanding the embodiments of this application, any person skilled in the art can make changes and modifications based on the technology taught in this application without departing from the spirit and scope of this application.

[0063] The illustrative embodiments and descriptions provided in this application are for explaining the application, but are not intended to limit the application. Furthermore, elements / components using the same or similar reference numerals in the drawings and embodiments are used to represent the same or similar parts.

[0064] The terms “first,” “second,” etc., used in this document are not intended to specifically refer to order or sequence, nor are they used to limit this application; they are merely used to distinguish elements or operations described using the same technical terms.

[0065] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0066] The term "and / or" as used herein includes any or all of the things mentioned.

[0067] The term "multiple" in this article includes "two" and "more than two"; the term "multiple groups" in this article includes "two groups" and "more than two groups".

[0068] Certain terms used to describe this application will be discussed below or elsewhere in this specification to provide additional guidance to those skilled in the art in describing the application.

[0069] Although research on malicious domain detection methods based on domain request association has been conducted for some time, their practical application remains difficult, mainly due to the following reasons:

[0070] (1) In the industry, DNS traffic detection devices are usually designed for large-scale data of billions. Many existing methods are not designed from an industrial perspective, so the time complexity and process are relatively complicated and difficult to implement.

[0071] (2) In the industry, there is very little labeled domain name data in the network data and the labels are usually time-sensitive. Supervised learning methods have great limitations, such as imbalance and label drift problems.

[0072] The DNS recursive resolution service records when a specific IP address accessed a given fully qualified domain name (FQDN), i.e.: Figure 1 As shown, a sample is represented as a triple (IP, time, FQDN): an IP accesses the domain name of FQDN1 at time1, and then accesses the domain name of FQDN2 at time2 very shortly after.

[0073] To solve the above problems, such as Figure 1 As shown, this application proposes a method for detecting accompanying domain names based on the locality-sensitive hashing algorithm, starting from the spatiotemporal association relationship of domain names and targeting large-scale DNS traffic resolution data of billions of records. The method includes the following steps:

[0074] S1. Calculate spatiotemporal adjoint pairs using the locality-sensitive hashing algorithm, and filter out the intersecting spatiotemporal adjoint pairs. The specific process is as follows:

[0075] S11. Coarse screening is performed based on time pairs (time, FQDN), and time adjoint pairs are calculated. The specific process is as follows:

[0076] S111, Calculate the time vector:

[0077] First, the millisecond-level timestamp (time) is converted into a second-level time index based on the time tuple (time, FQDN).

[0078] For example, a day has 24 hours and 86,400 seconds, and its second-level time index is [1:86400].

[0079] Then, extract the second-level time index corresponding to the time a domain name appears in a day, assign the value 1 to the corresponding index position, and build the time vector of the domain name.

[0080] For example, an FQDN domain name is represented as a time vector [0, 1, 0, 1, 0, 0], indicating that the FQDN domain name appeared at the 2nd second and the 4th second.

[0081] To save memory space and improve computational efficiency, dense vectors are usually sparsified before storage.

[0082] For example, the time vector [0, 1, 0, 1, 0, 0] in the above example can be sparsed and stored as sparse_vector(6, [2, 4]), where 6 represents 6 moments, 2 represents that the FQDN domain name appeared in the 2nd second, and 4 represents that the FQDN domain name appeared in the 4th second.

[0083] S112. Calculate the minimum hash fingerprint vector using the time vector:

[0084] Calculate the Jaccard distance between any two time vectors. The Jaccard distance can be expressed as a bit-wise AND / OR operation.

[0085] The Jaccard distance is approximated using the minimum hash function MinHash, yielding a low-dimensional approximate hash fingerprint for each high-dimensional vector. The mathematical relationship between the similarity between the minimum hash function MinHash and the Jaccard distance is expressed as follows:

[0086] (1)

[0087] In equation (1), and Represent two sets of [0-1] vectors, This indicates taking the intersection. To represent taking the union, pr Represents probability. Representing vectors and The similarity.

[0088] For example, as shown in Table 1, the minimum hash fingerprint is extracted from the three columns of numerical pairs [1, 1, 0, 0, 0, 1, 1] generated by three different hash functions, and the minimum hash fingerprint vector is [2, 2, 1].

[0089] Table 1. Illustration of MinHash function

[0090]

[0091] The specific process for extracting the minimum hash fingerprint is as follows:

[0092] In the first column of Table 1, the smallest value corresponding to 1 in vector V is found in the order of 1-7, and this value is used as the first element in the minimum hash fingerprint vector.

[0093] In the second column of Table 1, find the smallest value corresponding to 1 in vector V in the order of 1-7, and use this value as the second element in the minimum hash fingerprint vector.

[0094] In the third column of Table 1, find the smallest value corresponding to 1 in vector V in the order of 1-7, and use this value as the third element in the minimum hash fingerprint vector.

[0095] By establishing a minimum hash fingerprint, high-dimensional sparse vectors are approximately sampled and projected into a low-dimensional space, while also preserving the Jaccard distance between pairwise [0-1] vectors.

[0096] S113, Calculate time pair:

[0097] After the minimum hash function mapping, each high-dimensional 0-1 vector becomes a low-dimensional dense fingerprint vector. For example, the minimum hash fingerprint vector [2, 2, 1] obtained in step S112.

[0098] Locality-Sensitive Hashing (LSH) is used to divide a low-dimensional dense fingerprint vector of dimension L (i.e., a minimum hash fingerprint vector) into b bands, each band containing r rows.

[0099] (2)

[0100] In equation (2), L This represents the dimension of the minimum hash fingerprint vector. b This represents the number of band blocks in the minimum hash fingerprint vector. r This indicates the number of rows in each band.

[0101] like Figure 2 As shown, assuming there are n buckets, local vectors mapped to the same bucket are more similar. A band block of the third dense fingerprint vector and a band block of the fourth dense fingerprint vector are mapped to the same bucket 3. Therefore, a band block of the third dense fingerprint vector and a band block of the fourth dense fingerprint vector are candidate similarity pairs.

[0102] Assuming two dense fingerprint vectors and If the similarity between the two Jaccard distances is s, then according to the Locality Sensitive Hashing algorithm, the expected similarity threshold can be obtained. t for:

[0103] ;

[0104] in, Indicates all of the band blocks r The probability that all rows are equal. This indicates the probability that at least one row is different. This represents the probability that all bands are different. This represents the probability that at least one band is equal.

[0105] Therefore, the probability of each dense fingerprint vector being mapped into a bucket can be adjusted by setting the desired similarity threshold t and the three parameters L, b, and r.

[0106] This application ingeniously transforms the problem of calculating pairwise dense fingerprint vectors into a probabilistic problem, obtaining temporal adjoint pairs in sublinear time complexity.<fqdn_i,fqdn_j,s> Where fqdn_i represents the i-th fqdn domain name, and fqdn_j represents the j-th fqdn domain name.

[0107] Based on the obtained time-pair pair<fqdn_i,fqdn_j,s> If the similarity s is greater than the similarity threshold, then... t The time-coupling pair is the final time-coupling pair.

[0108] For example, the obtained time-coupling pair is<A1,B1,0.2> ,<C1,D1,0.6> If time pairs with a similarity s greater than 0.5 are taken as the final time pairs, then the final time pairs are:<C1,D1,0.6> .

[0109] S12. Based on the FQDN after initial screening using time association and the corresponding IP tuple (IP, time), a fine screening is performed to calculate spatial association pairs. The specific process is as follows:

[0110] Based on the time-coupling domain name obtained in step S11, extract the IP tuple (IP, time) and calculate the spatial coupling pair.

[0111] S121. Calculate spatial vectors:

[0112] First, the number of IPs is used as the IP index based on the spatial tuple (IP, FQDN).

[0113] For example, if the number of IPs appearing in a day is 10,000, its IP index is [1:10000].

[0114] Then, extract the IP indexes that appear in a day, assign the corresponding index position a value of 1, and establish the spatial vector of the domain name.

[0115] For example, an FQDN domain name is represented as a space vector [0, 1, 0, 1, 0, 0], indicating that the FQDN domain name appeared in the 2nd and 4th IP addresses.

[0116] To save memory space and improve computational efficiency, dense vectors are usually sparsified before storage.

[0117] For example, the spatial vector [0, 1, 0, 1, 0, 0] in the above example can be sparsed and stored as sparse_vector(6, [2, 4]), where 6 represents 6 IPs, 2 represents that the FQDN domain name appeared in the 2nd IP, and 4 represents that the FQDN domain name appeared in the 4th IP.

[0118] S122. Calculate the minimum hash fingerprint vector based on the spatial vector:

[0119] Calculate the Jaccard distance between any two spatial vectors. The Jaccard distance can be expressed as a bit-wise AND / OR operation.

[0120] The Jaccard distance is approximated using the minimum hash function MinHash, yielding a low-dimensional approximate hash fingerprint for each high-dimensional vector. The mathematical relationship between the similarity between the minimum hash function MinHash and the Jaccard distance is expressed as follows:

[0121] (1)

[0122] In equation (1), and Represents two sets of spatial vectors. This indicates taking the intersection. To represent taking the union, pr Represents probability. Representing vectors and The similarity.

[0123] For example, as shown in Table 2, the minimum hash fingerprint of the space vector [1, 1, 0, 1, 0, 1, 1] is extracted using three columns of values ​​generated by three different hash functions, and the minimum hash fingerprint vector is [2, 2, 1].

[0124] Table 2. Illustration of MinHash function

[0125]

[0126] The specific process for extracting the minimum hash fingerprint is as follows:

[0127] In the first column of Table 1, the smallest value corresponding to 1 in vector V is found in the order of 1-7, and this value is used as the first element in the minimum hash fingerprint vector.

[0128] In the second column of Table 1, find the smallest value corresponding to 1 in vector V in the order of 1-7, and use this value as the second element in the minimum hash fingerprint vector.

[0129] In the third column of Table 1, find the smallest value corresponding to 1 in vector V in the order of 1-7, and use this value as the third element in the minimum hash fingerprint vector.

[0130] By establishing a minimum hash fingerprint, high-dimensional sparse vectors are approximately sampled and projected into a low-dimensional space, while also preserving the Jaccard distance between pairwise spatial vectors.

[0131] S123. Based on the minimum hash fingerprint vector, the spatial adjoint pair is calculated using the locality-sensitive hashing algorithm:

[0132] After the minimum hash function mapping, each high-dimensional 0-1 vector becomes a low-dimensional dense fingerprint vector. For example, the minimum hash fingerprint vector [2, 2, 1] obtained in step S112.

[0133] Locality-Sensitive Hashing (LSH) is used to divide a low-dimensional dense fingerprint vector of dimension L (i.e., a minimum hash fingerprint vector) into b bands, each band containing r rows.

[0134] (3)

[0135] In equation (3), L This represents the length of the minimum hash fingerprint vector. b This represents the number of band blocks in the minimum hash fingerprint vector. r This indicates the number of rows in each band.

[0136] like Figure 2 As shown, assuming there are n buckets, local vectors mapped to the same bucket are more similar. A band block of the third dense fingerprint vector and a band block of the fourth dense fingerprint vector are mapped to the same bucket 3. Therefore, a band block of the third dense fingerprint vector and a band block of the fourth dense fingerprint vector are candidate similarity pairs.

[0137] Assuming two dense fingerprint vectors and If the similarity between the two Jaccard distances is s, then according to the Locality Sensitive Hashing algorithm, the expected similarity threshold can be obtained. t for:

[0138] ;

[0139] in, Indicates all of the band blocks r The probability that all rows are equal. This indicates the probability that at least one row is different. This represents the probability that all bands are different. This represents the probability that at least one band is equal.

[0140] Therefore, the probability of each dense fingerprint vector being mapped into a bucket can be adjusted by setting the desired similarity threshold t and the three parameters L, b, and r.

[0141] This application ingeniously transforms the problem of calculating pairwise dense fingerprint vectors into a probabilistic problem, obtaining spatial adjoint pairs in sublinear time complexity.<fqdn_u,fqdn_v,s> Where fqdn_u represents the u-th fqdn domain name, and fqdn_v represents the v-th fqdn domain name.

[0142] Based on the obtained spatial pair<fqdn_u,fqdn_v,s> If the similarity s is greater than the similarity threshold, then... t The spatial companion pair is the final spatial companion pair.

[0143] For example, the resulting spatial adjoint pairs are<A2,B2,0.2> ,<C2,D2,0.6> Spatial adjoint pairs with a similarity s greater than 0.5 are taken as the final spatial adjoint pairs. Therefore, the final spatial adjoint pairs are:<C2,D2,0.6> .

[0144] S13, Accompany the obtained time with the pair<fqdn_i,fqdn_j,s> Accompanying space<fqdn_u,fqdn_v,s> Perform a JOIN operation to obtain the final intersection spatiotemporal adjoint pair.<fqdn_m,fqdn_k,s> Where fqdn_m represents the m-th fqdn domain name, and fqdn_k represents the k-th fqdn domain name.

[0145] S2. Calculate and sort the true C-Rank scores of the intersection spatiotemporal companion pairs, and provide a companion domain data service interface to facilitate the rapid search for companion domains of malicious domains. The specific process is as follows:

[0146] Assuming that any intersection of spacetime adjoint pairs is known<fqdn1,fqdn2,s> Then we can calculate:

[0147] The number of times the domain name fqdn1 appears in the data within one day is 'a'.

[0148] The domain name fqdn2 appears b times in the data for one day;

[0149] Within a certain time window (e.g., 5 minutes), if domain name fqdn1 appears, the number of times domain name fqdn2 appears is c.

[0150] Given that domain name fqdn2 appears within a certain time window (e.g., 5 minutes), the number of times domain name fqdn1 appears is d.

[0151] Within a certain time window (e.g., 5 minutes), the number of IPs that simultaneously access the intersection of domain names fqdn1 and fqdn2 is g;

[0152] According to Bayes' theorem, given that domain name fqdn1 appears, the probability of domain name fqdn2 appearing is... Given that domain name fqdn2 appears, the probability of domain name fqdn1 appearing is: .

[0153] Using the Kulc coefficient in statistics, the similarity between two intersecting spatiotemporal adjoint pairs can be expressed as:

[0154] (4)

[0155] when x and y When they are equal, the function The value is the largest.

[0156] Considering the adjoint intersection counts c and d, and the number of adjoint intersection IPs g, a smoothing function sigmoid is introduced to smooth the adjoint intersection counts c and d, and the number of adjoint intersection IPs g. Mathematically, this is expressed as:

[0157] (5)

[0158] In equation (5), This represents the smoothing coefficient. It can be seen from equation (4) that the function... It is proportional to the number of adjoint intersections c and d, and the number of adjoint intersection IPs g.

[0159] Finally, the C-Rank score of the intersection spacetime adjoint pair is:

[0160] (6)

[0161] In this way, the C-Rank score preserves similarity while being proportional to the number of adjoint intersections c and d, as well as the number of adjoint intersection IPs g.

[0162] S3. Based on the above-mentioned intersectional spatiotemporal pair<fqdn_m,fqdn_k,s> By grouping target domains, the companion pairs of target domains with a similarity s greater than the threshold t are taken as the final time companion pairs.

[0163] For example, for the accompanying pair<A,B,0.6> ,<A,C,0.7> ,<A,D,0.8> If the three companion pairs are grouped according to domain name A, and the first two companion pairs are selected based on similarity, then domain names C and D are chosen.

[0164] After the spatiotemporal pair calculation in step S1 above, the adjoint probability N can be... 2 Reduce to M candidate adjoint pairs of arbitrary size. Calculate the final spatiotemporal C-Rank score of the intersection based on three dimensions: similarity between the two intersecting spatiotemporal adjoint pairs, the number of adjoint intersections, and the number of IPs in the adjoint intersection.

[0165] The accompanying domain name detection method based on the Locality Sensitive Hash algorithm provided in this application embodiment can quickly calculate the accompanying domain name pairs in industrial-grade massive data scenarios (billions of DNS recursive resolution data) based on the domain name request companion relationship and using the Locality Sensitive Hash algorithm in linear time complexity.

[0166] In an exemplary embodiment, this application also provides a companion domain name detection system based on the locality-sensitive hash algorithm, used to complete the steps in the companion domain name detection method based on the locality-sensitive hash algorithm described above.

[0167] In an exemplary embodiment, this application also provides a computer storage medium, which is a computer-readable storage medium, such as a memory including a computer program. The computer program can be executed by a processor in a domain name detection system based on locality-sensitive hashing algorithm to perform the steps in the domain name detection method based on locality-sensitive hashing algorithm.

[0168] Computer-readable storage media can be magnetic random access memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, flash memory, magnetic surface memory, optical disc, or read-only optical disc, etc.

[0169] The above description is merely an illustrative embodiment of this application. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of this application shall fall within the scope of protection of this application.

Claims

1. A method for detecting accompanying domain names based on the locality-sensitive hashing algorithm, characterized in that, Includes the following steps: The locality-sensitive hashing algorithm is used to calculate spatiotemporal adjoint pairs and filter out the intersection spatiotemporal adjoint pairs. The specific process is as follows: Coarse screening is performed based on time pairs (FQDNs), and time-paired pairs are calculated. Based on the FQDN after initial screening using time association and the corresponding IP tuple (IP, time), a fine screening is performed to calculate spatial association pairs; The obtained temporal and spatial pairs are joined together to obtain the final intersection of the temporal and spatial pairs. Calculate and sort the true C-Rank scores of the spatiotemporal adjoint pairs in the intersection. The final spatiotemporal C-Rank score of the intersection is calculated based on three dimensions: the similarity between the two spatiotemporal adjoint pairs, the number of adjoint intersections, and the number of adjoint intersection IPs. By grouping the target domains, the companion pairs of target domains with a similarity greater than the similarity threshold are taken as the final time companion pairs; The specific process of coarse screening based on time tuples (FQDN) and calculating time adjoining pairs is as follows: Calculate the time vector, and use the time vector to calculate the minimum hash fingerprint vector; Based on the minimum hash fingerprint vector, the time-adjoint pair is calculated using the locality-sensitive hashing algorithm; The specific process for calculating the time vector is as follows: based on the time tuple (time, FQDN), the millisecond-level timestamp time is converted into a second-level time index; Extract the second-level time index corresponding to the time a domain name appears in a day, assign the value 1 to the corresponding index position, and build the time vector of the domain name; The specific process of calculating spatial adjoint pairs based on the FQDN after initial screening using time-adjoint pairs and the corresponding IP tuples (IP, time) is as follows: Calculate the spatial vector, and then calculate the minimum hash fingerprint vector based on the spatial vector. Based on the minimum hash fingerprint vector, spatial adjoint pairs are calculated using the locality-sensitive hashing algorithm; The specific process for calculating the spatial vector is as follows: the number of IPs is used as the IP index based on the spatial tuple (IP, FQDN); Extract the IP indexes that appear in a day, assign a value of 1 to the corresponding index position, and create a spatial vector for that domain name.

2. The accompanying domain name detection method based on the locality-sensitive hashing algorithm according to claim 1, characterized in that, The specific process of calculating the minimum hash fingerprint vector using the time vector is as follows: Calculate the Jaccard distance between any two time vectors; where the Jaccard distance is expressed as a bitwise AND operation or a bitwise OR operation. The Jaccard distance is approximated using the minimum hash function MinHash to obtain a low-dimensional approximate hash fingerprint for each high-dimensional vector.

3. The accompanying domain name detection method based on the locality-sensitive hash algorithm according to claim 2, characterized in that, The specific process of calculating the time-coupling pair based on the minimum hash fingerprint vector using the locality-sensitive hashing algorithm is as follows: Using the locality-sensitive hashing algorithm to divide the dimension L The low-dimensional dense fingerprint vector is divided into b bands, each band having r rows, where... ; Assuming two dense fingerprint vectors and If the similarity between the two Jaccard distances is s, then the expected similarity threshold is obtained according to the Locality Sensitive Hashing algorithm. t for ; Similarity s Greater than the similarity threshold t The time-coupling pair is the final time-coupling pair.

4. The accompanying domain name detection method based on the locality-sensitive hashing algorithm according to claim 1, characterized in that, The specific process for calculating the minimum hash fingerprint vector based on the spatial vector is as follows: Calculate the Jaccard distance between any two spatial vectors, where the Jaccard distance is expressed as a bitwise AND operation or a bitwise OR operation; The Jaccard distance is approximated using the minimum hash function MinHash to obtain a low-dimensional approximate hash fingerprint for each high-dimensional vector.

5. The accompanying domain name detection method based on the locality-sensitive hashing algorithm according to claim 4, characterized in that, The specific process of calculating the spatial adjoint pair based on the minimum hash fingerprint vector using the locality-sensitive hashing algorithm is as follows: Using the locality-sensitive hashing algorithm to divide the dimension L The low-dimensional dense fingerprint vector is divided into b bands, each band having r rows, where... ; Assuming two dense fingerprint vectors and If the similarity between the two Jaccard distances is s, then the expected similarity threshold is obtained according to the Locality Sensitive Hashing algorithm. t for ; Similarity s Greater than the similarity threshold t The temporal companion pair serves as the final spatial companion pair.

6. The accompanying domain name detection method based on the locality-sensitive hashing algorithm according to claim 1, characterized in that, The specific process of calculating and sorting the true C-Rank scores of the intersection spatiotemporal adjoint pairs is as follows: Assuming that any intersection of spacetime adjoint pairs is known<fqdn1,fqdn2,s> Let s represent the similarity, then the following can be calculated: The number of times the domain name fqdn1 appears in the data within one day is 'a'. The domain name fqdn2 appears b times in the data for one day; Within a certain time window, given that domain name fqdn1 appears, the number of times domain name fqdn2 appears is c. Within a certain time window, if domain name fqdn2 appears, the number of times domain name fqdn1 appears is d; Within a certain time window, the number of IPs that simultaneously access the intersection of domain names fqdn1 and fqdn2 is g; According to Bayes' theorem, given that domain name fqdn1 appears, the probability of domain name fqdn2 appearing is... ; Given that domain name fqdn2 exists, the probability of domain name fqdn1 existing is: ; Using the Kulc coefficient in statistics, the similarity between two intersecting spatiotemporal adjoint pairs can be expressed as: ， when x and y When they are equal, the function The value is the largest; The sigmoid function is introduced to smooth the adjoint intersection counts c and d, as well as the number of adjoint intersection IPs g. Mathematically, it is expressed as: ， In the formula, Indicates the smoothing coefficient; Finally, the C-Rank score of the intersection spacetime adjoint pair is: 。

Images