Statistical method and device for DNS (Domain Name Server) data

Abstract

The invention discloses a statistical method and device for DNS (Domain Name Server) data. When the method provided by the embodiment of the invention is executed, firstly, the DNS logs are traversed, the source IP and the query domain name of each query log are extracted, and the update field is constructed on the basis of the source IP and the query domain name. Hash calculation is performed on the source IP in the update field to generate a first Hash result, so that a corresponding calculation unit is determined. The calculation unit belongs to a composite data structure and comprises a plurality of super logarithm counting units and a counting minimum sketch. Then, the query domain name in the update field is used as a keyword to update the calculation unit, and an updated composite data structure is obtained; and finally, performing data statistics in the updated data structure according to statistical requirements. Through Hash calculation and a composite data structure, frequent traversal of the whole log file can be reduced, so that calculation resources and time are saved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a DNS data statistics method and device. Background Art

[0002] In today's digital age, the stability and performance of network services are crucial to both businesses and users. As one of the infrastructures of the Internet, the Domain Name System (DNS) undertakes the important task of mapping human-readable domain names to computer-understandable Internet Protocol (IP) addresses. This process is not only the basis for users to access websites, but also a key link in the normal operation of various Internet services. Therefore, in order to ensure the stability and reliability of DNS services, it is crucial to monitor and analyze DNS log data.

[0003] DNS logs record query requests from DNS authoritative servers or recursive servers, including key information such as the source IP address and requested domain name. By collecting statistics and analyzing these DNS log data, network administrators can gain in-depth insights into the use of DNS services, identify potential problems in a timely manner, and take appropriate measures to optimize and improve them. For example, counting the total number of requests can help evaluate server load and performance, identify traffic peaks and normal usage patterns, which is crucial for resource allocation and service quality improvement. In addition, analyzing the distribution of requested domain names can reveal hot domain names, popular services, and potential security threats, thereby providing data support for security protection.

[0004] However, although monitoring DNS log status is a core task in network management, existing DNS log data processing methods still face some challenges and limitations. One major challenge is that frequent query tasks usually lead to huge resource consumption. This is mainly because the DNS log data volume is huge, and when performing multiple data statistics tasks, it is often necessary to traverse the entire log content multiple times, which consumes a lot of time and computing resources.

[0005] Therefore, how to reduce the consumption of computing resources by DNS logs when performing multiple data statistics and other tasks, while improving the efficiency of data statistics, is a technical problem that technicians in this field urgently need to solve. Summary of the invention

[0006] Based on the above problems, the present application provides a DNS data statistics method and device, which can reduce the consumption of computing resources by DNS logs when performing multiple data statistics tasks, while improving the efficiency of data statistics.

[0007] The embodiments of the present application disclose the following technical solutions:

[0008] A DNS data statistics method, the method comprising:

[0009] Traversing a domain name system (DNS) log to determine one or more query logs in the DNS log;

[0010] Determine the corresponding source Internet Protocol IP of each query log and the query domain name in each query log, and construct an update field for each query log based on the source IP and the query domain name of each query log; the update field includes the source IP and the query domain name;

[0011] The source IP in the update field is used as a keyword to perform a hash calculation to obtain a first hash result, and a calculation unit corresponding to the update field is determined based on the first hash result; the DNS where the DNS log is located includes a composite data structure, the composite data structure includes a plurality of the calculation units, each of the calculation units includes a superlogarithmic counting unit and a counting minimum sketch;

[0012] Using the query domain name in the update field as a keyword, updating the superlogarithmic counting unit and the counting minimum sketch of the calculation unit corresponding to the update field, to obtain an updated composite data structure;

[0013] Based on statistical requirements, data statistics are performed in the updated composite data structure.

[0014] In a possible implementation, the method further includes:

[0015] Initialize the superlogarithmic counting unit:

[0016] Randomly select a parameter p, and calculate the number of buckets m according to the parameter p; m = 2^p;

[0017] Create a super logarithmic array of size m and value 0; the super logarithmic array is represented as M[0]...M[m-1];

[0018] Initialize the counting minimum sketch:

[0019] Randomly determine the width w and depth d;

[0020] Initialize a two-dimensional array sketch of size d*w, where the initial value of each element in sketch is 0; the two-dimensional array sketch is represented by sketch[0...d-1]...[0...w-1];

[0021] Randomly generate d independent hash functions h0, h1, ..., hd-1, and map each hash function to the range of [0, w-1] to obtain mapped hash functions h0′, h1′, ..., hd-1′; the d hash functions have a one-to-one correspondence with each column of the two-dimensional array sketch.

[0022] In a possible implementation manner, using the query domain name in the update field as a keyword to update the superlogarithm counting unit of the calculation unit corresponding to the update field includes:

[0023] Performing a hash calculation using the query domain name in the update field as a keyword to obtain a second hash result;

[0024] Determine a first index value based on the first p bits of the second hash result, and calculate the number of leading zeros rho(w) of the bits other than the first p bits of the second hash result; the first index value j=h(x)[0:p-1]&(m-1); h(x)[0:p-1] is the first p bits of the second hash result; h(x) is the second hash result;

[0025] Update the M[j]th bucket in the superlogarithm array based on the first index value j and the number of leading zeros rho(w) to obtain an updated superlogarithm array;

[0026] Among them, the updated M[j]th bucket is max(M[j], rho(w)).

[0027] In a possible implementation, using the query domain name in the update field as a keyword to update the count minimum sketch of the computing unit corresponding to the update field includes:

[0028] The query domain name in the update field is input as a keyword into the mapped hash functions h0′, h1′, ..., hd-1′ to perform hash calculations, and d second index values ​​h0′(x), h1′(x), ..., hd-1′(x) are obtained; x is the query domain name in the update field; and the d second index values ​​have a one-to-one correspondence with each column of the two-dimensional array sketch;

[0029] Based on d second index values ​​and the columns of the two-dimensional array sketch corresponding to the d second index values, d items to be calculated are determined from the two-dimensional array sketch; the second index values ​​are used to determine the rows in the two-dimensional array sketch where the items to be calculated are located, and the columns in the two-dimensional array sketch corresponding to the second index values ​​are used to determine the columns in the two-dimensional array sketch where the items to be calculated are located;

[0030] The d items to be calculated in the two-dimensional array sketch are respectively increased by 1 to obtain an updated sketch with minimum count.

[0031] In a possible implementation, the statistical requirements include:

[0032] One or more of counting the total number of queries of all source IPs in the DNS log, counting the query volume of the first target source IP in the DNS log, counting the total query volume of the first target query domain name in the DNS log, counting the query cardinality of the second target source IP in the DNS log, and counting the query volume of the query domain name corresponding to the third target source IP in the DNS log;

[0033] Among them, the query domain name corresponding to the third target source IP and the third target source IP are in the same update field.

[0034] In a possible implementation, in the updated composite data structure, the total number of queries from all source IPs in the DNS log is counted, including:

[0035] The two-dimensional arrays of the minimum count sketches of all the calculation units in the updated composite data structure are all used as the array to be calculated;

[0036] Adding the first column numbers of each of the arrays to be calculated respectively to obtain the addition results of each of the arrays to be calculated;

[0037] The addition results of each of the arrays to be calculated are added together to obtain the total number of queries from all source IPs.

[0038] In a possible implementation, in the updated composite data structure, counting the query volume of the first target source IP in the DNS log includes:

[0039] Performing hash calculation using the first target source IP as a keyword to obtain a third hash result;

[0040] In the updated composite data structure, determining the computing unit corresponding to the first target source IP as the first target computing unit based on the third hash result;

[0041] The first column numbers of the two-dimensional array sketch of the minimum counting sketch of the first target computing unit are added to obtain the query volume of the first target source IP.

[0042] In a possible implementation, in the updated composite data structure, counting the total query volume of the first target query domain name in the DNS log includes:

[0043] querying the one or more update fields where the first target query domain name is located, and determining the one or more update fields where the first target query domain name is located as target fields;

[0044] Performing hash calculation on the source IP in each of the target fields to obtain one or more fourth hash results, and determining one or more computing units corresponding to the first target query domain name as second target computing units based on the fourth hash results;

[0045] Calculate and add the frequency estimation values ​​of the minimum counting sketches corresponding to the first target query domain name in each of the second target calculation units, and use the sum of the frequency estimation values ​​as the total query volume of the first target query domain name in the DNS log;

[0046] Among them, the calculation process of the frequency estimation value of the counting minimum sketch corresponding to the first target query domain name in each of the second target computing units includes: inputting the first target query domain name into the hash functions h0′, h1′, ..., hd-1′ mapped by the counting minimum sketch of the second target computing unit for hash calculation, obtaining d third index values ​​h0′(y), h1′(y), ..., hd-1′(y), and taking the minimum value of the d third index values ​​as the frequency estimation value of the counting minimum sketch corresponding to the first target query domain name in the second target computing unit; y is the first target query domain name.

[0047] In a possible implementation, in the updated composite data structure, counting the query cardinality of the second target source IP in the DNS log includes:

[0048] Performing hash calculation using the second target source IP as a keyword to obtain a fifth hash result;

[0049] In the updated composite data structure, determining the computing unit corresponding to the second target source IP as the third target computing unit based on the fifth hash result;

[0050] Calculate the cardinality estimate of the second target source IP in the superlogarithm counting unit of the third target computing unit as the query cardinality of the second target source IP;

[0051] Among them, the cardinality estimate E=alpha*m^2*Z; alpha is the calibration constant; m is the number of buckets of the superlogarithmic counting unit of the third target computing unit; Z is the average value, Z=(1 / m)*sum(2^(-M[i])), i=0,...,m-1; M[i] is the element in the superlogarithmic array of the superlogarithmic counting unit of the third target computing unit.

[0052] In a possible implementation, in the updated composite data structure, counting the query volume of the query domain name corresponding to the third target source IP in the DNS log includes:

[0053] Performing a hash calculation using the third target source IP as a keyword to obtain a sixth hash result;

[0054] In the updated composite data structure, determining the computing unit corresponding to the third target source IP as the fourth target computing unit based on the sixth hash result;

[0055] Calculate the frequency estimation value of the query domain name corresponding to the third target source IP in the counting minimum sketch of the fourth target calculation unit as the query amount of the query domain name corresponding to the third target source IP;

[0056] Among them, the calculation process of the frequency estimation value of the query domain name corresponding to the third target source IP in the counting minimum sketch of the fourth target computing unit includes: inputting the query domain name corresponding to the third target source IP into the hash functions h0′, h1′, ..., hd-1′ mapped by the counting minimum sketch of the fourth target computing unit for hash calculation, obtaining d fourth index values ​​h0′(q), h1′(q), ..., hd-1′(q), and taking the minimum value of the d fourth index values ​​as the frequency estimation value of the query domain name corresponding to the third target source IP in the counting minimum sketch of the fourth target computing unit; q is the query domain name corresponding to the third target source IP.

[0057] A statistical device for DNS data, the device comprising:

[0058] A first determination module is used to traverse a domain name system DNS log and determine one or more query logs in the DNS log;

[0059] A second determination module is used to determine the corresponding source Internet Protocol IP of each query log and the query domain name in each query log, and to construct an update field for each query log based on the source IP and the query domain name of each query log; the update field includes the source IP and the query domain name;

[0060] A first calculation module, configured to perform a hash calculation using the source IP in the update field as a keyword to obtain a first hash result;

[0061] A third determination module is used to determine the calculation unit corresponding to the update field based on the first hash result; the DNS where the DNS log is located includes a composite data structure, the composite data structure includes a plurality of the calculation units, each of the calculation units includes a superlogarithmic counting unit and a counting minimum sketch;

[0062] An updating module, configured to use the query domain name in the update field as a keyword to update the superlogarithmic counting unit and the counting minimum sketch of the calculation unit corresponding to the update field, so as to obtain an updated composite data structure;

[0063] The data statistics module is used to perform data statistics in the updated composite data structure based on statistical requirements.

[0064] Compared with the prior art, this application has the following beneficial effects:

[0065] The present application provides a statistical method and device for DNS data. Specifically, when executing the statistical method for DNS data provided by the embodiment of the present application, the DNS log can be first traversed to determine one or more query logs in the DNS log, and the corresponding source IP of each query log and the query domain name in each query log are determined, so as to construct update fields for each query log based on the source IP and query domain name of each query log. The source IP in the update field is used as a keyword to perform a hash calculation to obtain a first hash result, and the calculation unit corresponding to the update field is determined based on the first hash result. The DNS where the DNS log is located includes a composite data structure, and the composite data structure includes multiple calculation units, each of which includes a superlogarithmic counting unit and a counting minimum sketch. Then, the query domain name in the update field is used as a keyword to update the calculation unit corresponding to the update field to obtain an updated composite data structure. Finally, based on the statistical requirements, data statistics are performed in the updated composite data structure. The present application reduces multiple traversals of the entire log content through the use of hash calculations and composite data structures, thereby reducing computing resources and time consumption. At the same time, the composite data structure combined with the hyperlogarithmic counting unit and the counting minimum sketch can effectively process large amounts of data and provide accurate statistical results, which is suitable for high-concurrency query environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 A method flow chart of a DNS data statistics method provided in an embodiment of the present application;

[0068] Figure 2 A method flow chart of a superlogarithmic counting unit updating method provided in an embodiment of the present application;

[0069] Figure 3 A method flow chart of a counting minimum sketch updating method provided in an embodiment of the present application;

[0070] Figure 4 A method flow chart of a method for counting the total number of queries from all source IPs in a DNS log provided in an embodiment of the present application;

[0071] Figure 5 A method flow chart of a method for counting query volume of a target source IP in a DNS log provided in an embodiment of the present application;

[0072] Figure 6 A method flow chart of a method for counting the total query volume of a target query domain name in a DNS log provided in an embodiment of the present application;

[0073] Figure 7 A method flow chart of a method for counting query cardinality of a target source IP in a DNS log provided in an embodiment of the present application;

[0074] Figure 8 A method flow chart of a method for counting query volume of a query domain name corresponding to a target source IP in a DNS log provided in an embodiment of the present application;

[0075] Fig. 9 A schematic diagram of the structure of a DNS data statistics device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the background technology involved in the embodiments of the present application will be described below.

[0077] In today's digital age, the stability and performance of network services are of vital importance to both enterprises and users. As one of the Internet infrastructures, DNS plays a key role in mapping easy-to-read domain names to computer-understandable IP addresses. Ensuring the reliability and stability of DNS services has become an inevitable choice. Monitoring and analyzing DNS log data has become indispensable. These logs record the query request information received by the authoritative server or recursive server, including key data such as the source IP and the requested domain name. Through the statistics and analysis of DNS log data, administrators can gain insight into the use of DNS services, find problems in a timely manner, and take corresponding optimization measures.

[0078] Monitoring DNS log status is one of the core tasks of network management. Counting the total number of requests helps evaluate server load and performance, and gain insight into traffic peaks and normal patterns; analyzing the distribution of requested domain names reveals popular domain names, popular services, and potential security risks. With these data statistics, administrators can better understand the operating status of DNS services and make adjustments and policy improvements as needed.

[0079] Monitoring and analyzing DNS log data is an indispensable step in network management and operation and maintenance, but there are some challenges and limitations. The main challenge is that frequent query tasks lead to huge resource consumption. DNS log data is huge, and when performing multiple statistical and analysis tasks, multiple queries need to traverse the entire log content, which will take up a lot of time and computing resources.

[0080] In order to solve this problem, a statistical method and device for DNS data are provided in an embodiment of the present application. First, the DNS log is traversed to identify one or more query records, and the source IP and query domain name of each record are extracted, and then an update field is constructed for each record based on this information. Next, the source IP of the update field is used as a keyword for hash calculation, and the first hash result is generated to determine the corresponding calculation unit. In this composite data structure, each calculation unit contains a superlogarithmic counting unit and a counting minimum sketch. Subsequently, the query domain name of the update field is used as a keyword to update the calculation unit, thereby obtaining an updated composite data structure. Finally, data analysis is performed in this structure according to statistical requirements. The present application helps to reduce the number of traversals of the entire DNS log through hash calculations and composite data structures, thereby improving the data processing speed. At the same time, the use of superlogarithmic counting units and counting minimum sketches can perform large-scale data statistics at a lower consumption of computing resources, which is suitable for processing massive DNS logs.

[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0082] See also Figure 1 , which is a flow chart of a DNS data statistics method provided by an embodiment of the present application, such as Figure 1 As shown, the DNS data statistics method may include steps S101-S105:

[0083] S101: Traverse a Domain Name System (DNS) log to determine one or more query logs in the DNS log.

[0084] Traversing the DNS log to identify one or more query logs in the DNS log refers to the process of accessing and parsing the DNS log file that stores the domain name resolution activity records. The goal of this process is to filter out specific DNS query events from a large amount of DNS log information.

[0085] "Traversal" means checking each record in the DNS log file one by one. In each log record, find those records that actually contain DNS query requests. "Determine" means identifying and extracting those records that meet specific conditions (i.e. query operations).

[0086] It should be noted that the query log in the DNS log is a log file that records the Domain Name System (DNS) query requests. It contains detailed information about the query requests sent by users or systems to the DNS server when accessing the Internet. These logs contain various key data points to help network administrators, system operators and network security analysts understand various aspects of domain name system activities. The main contents of the query log can include but are not limited to:

[0087] Query initiation time: Indicates when the query was sent to the DNS server.

[0088] Queryer IP address: The IP address of the device or user performing the query.

[0089] Query type: The target type of the DNS query request, such as A (Internet Protocol version 4 (IPv4) address), AAAA (Internet Protocol version 6 (IPv6) address), Mail Exchange Record (MX), Canonical Name Record (CNAME), etc.

[0090] Query domain name: The domain name that the user is trying to resolve, that is, the queried domain name, such as "www.example.com"

[0091] Query results: whether the query is successfully parsed, the type of records parsed (if any are returned), and the results of the parsing (such as IP address).

[0092] Query response time: The time it takes for the DNS server to respond to a query, as well as the total time of the entire query process.

[0093] Response code: The HyperText Transfer Protocol (HTTP) status code of the DNS response. For example, 0 represents success, 3 represents a non-existent domain name, and 4 represents no resolver providing the service.

[0094] Abnormal events: If the query fails or an error occurs, the specific error information can be recorded in detail.

[0095] S102: Determine the corresponding source Internet Protocol IP of each query log and the query domain name in each query log, and construct an update field for each query log based on the source IP and the query domain name of each query log.

[0096] Step S102 mainly involves extracting and constructing key information from the DNS query log. Specifically, each query log needs to be parsed first to determine the IP address of its source, which refers to the unique network identifier of the device or user that initiated the query request. At the same time, the domain name information involved in the query log must be extracted, that is, the name of the specific website or service requested by the user, such as "www.example.com". After completing the extraction of these two pieces of information, an "update field" will be constructed for each log based on this information. This update field is a data structure specifically used to store and represent the relationship between the source IP and the query domain name. By constructing such an update field, subsequent data processing and statistical analysis can be facilitated, because the updateable field organizes key query information together, providing a basis for further operations. This method helps to efficiently manage and analyze large amounts of DNS query data, thereby supporting subsequent decision-making and optimization.

[0097] In a possible implementation, the updated field is in the form of <source IP, query domain name>.

[0098] S103: Perform a hash calculation using the source IP in the update field as a keyword to obtain a first hash result, and determine a calculation unit corresponding to the update field based on the first hash result.

[0099] Step S103 mainly involves the process of processing and analyzing DNS logs through hash calculations. Specifically, first, the source IP address in the extracted update field is used as a keyword for hash calculation, and the result of this hash calculation is called the "first hash result". This result is used to determine the calculation unit corresponding to the update field. The data structure stored in the DNS log is composite, which means that it is composed of multiple components. In this composite data structure, there are multiple calculation units, each of which contains two main counting mechanisms: hyperlogarithmic counting units and counting minimum sketches. Hyperlogarithmic counting units are generally used to efficiently estimate the frequency or statistics of certain events, while counting minimum sketches are used for traffic statistics with higher memory efficiency. The design of this structure allows for fast and efficient processing and analysis of a large number of DNS queries, thereby supporting deeper data mining and network monitoring.

[0100] In a possible implementation, the process of performing hash calculation includes: when performing hash calculation, it is necessary to first select a suitable hash function, such as MD5, SHA-1, SHA-256, etc. The hash function should have the characteristics of fast calculation, uniform distribution and anti-collision. Then determine the data to be hashed, and use the selected hash function to process the data to be hashed to generate a hash value of fixed length. The obtained hash value is usually expressed in binary form and can be used for subsequent data processing and storage.

[0101] It should be noted that HyperLogLog (HLL) is a very efficient algorithm for estimating the number of elements in large-scale data sets. Its main feature is that it can estimate the size of a set with a relatively small amount of memory space and provides a good indicator of estimation uncertainty. The principle of HLL is based on the relationship between random variables and the maximum value. A set of random numbers is generated by introducing a hash function to estimate the number of elements in a set. The advantage of HLL is that it can achieve high accuracy while maintaining relatively low memory consumption.

[0102] Count-Min Sketch is another statistical method for storing and querying probability distribution. It consists of multiple count value matrices, and the value of each element represents the number of times it has been inserted at this position. When the count value of an element needs to be queried, the minimum value of this element in all matrices is used as the final estimate. A key feature of this method is that it can reduce the space cost, but it will sacrifice some accuracy. At the same time, it can support online updates and concurrent operations, so it is very useful in big data processing scenarios.

[0103] It should also be noted that the superlogarithmic counting unit and the counting minimum sketch in each calculation unit have been initialized.

[0104] In a possible implementation manner, initializing the superlogarithmic counting unit includes:

[0105] First, a parameter p is randomly selected, and the number of buckets m is calculated according to the parameter p, where m = 2^p. Then, a hyperlogarithmic array of size m and value 0 is created; the hyperlogarithmic array is represented as M[0]...M[m-1].

[0106] It should be noted that in the algorithm of the superlogarithmic counting unit, a "bucket" is a data structure used to store statistical information of data within a specific range to help calculate the approximate number of independent elements.

[0107] In a possible implementation, initializing the counting minimum sketch includes:

[0108] First, randomly determine the width w and depth d, and then initialize a two-dimensional array sketch of size d*w, where the initial value of each element in sketch is 0; the two-dimensional array sketch is represented by sketch[0...d-1]...[0...w-1]. Then randomly generate d independent hash functions h0, h1,..., h d-1 And map each hash function to the range of [0,w-1] to obtain the mapped hash functions h0′, h1′, ..., h d-1 ′.

[0109] The d hash functions have a one-to-one correspondence with each column of the two-dimensional array sketch, that is, the hash function h0 corresponds to the first column of the two-dimensional array sketch, h1 corresponds to the second column of the two-dimensional array sketch, and so on. d-1 Corresponding to the dth column of the two-dimensional array sketch. Similarly, the mapped hash functions h0′, h1′, ..., h d-1 ′ also has a one-to-one correspondence with each column of the two-dimensional array sketch.

[0110] In a possible implementation, the first hash result is a binary number, and the decimal value range of the first hash result is [0, the number of computing units]. The number of computing units is the number of all computing units included in the composite data structure.

[0111] It should be noted that all hash results in this application are binary numbers.

[0112] S104: Using the query domain name in the update field as a keyword, the superlogarithmic counting unit and the counting minimum sketch of the calculation unit corresponding to the update field are updated to obtain an updated composite data structure.

[0113] Using the query domain name in the update field as a keyword means using the domain name to find the corresponding calculation unit in the composite data structure for update. This process involves modifying the superlogarithmic counting unit and the counting minimum sketch within the calculation unit. Specifically, for the query domain name of each query log, the system uses the query domain name as a keyword to locate the corresponding calculation unit. Within this calculation unit, the system will update the superlogarithmic counting unit and the counting minimum sketch to reflect the number of occurrences of the query domain name or other related information. Such an update process helps maintain the accuracy and integrity of the data structure, while allowing the system to dynamically record and track statistical information for different query domain names.

[0114] In general, step S104 aims to update the data structure in the calculation unit according to the query domain name, so as to achieve in-depth analysis and recording of the DNS log data. By using the query domain name as a keyword for updating, the system can effectively manage and maintain the counting information related to different domain names and provide a basis for further statistical analysis. Therefore, this process is to ensure that the data structure is updated and accurately reflects the frequency or other relevant indicators of each query domain name in the DNS log.

[0115] See also Figure 2 , Figure 2 A method flow chart of a method for updating a superlogarithmic counting unit provided in an embodiment of the present application. Accordingly, using the query domain name in the update field as a keyword to update the superlogarithmic counting unit of the calculation unit corresponding to the update field can be implemented by steps A1-A3:

[0116] A1: Perform a hash calculation using the query domain name in the update field as a keyword to obtain a second hash result.

[0117] In order to update the super-logarithmic counting unit, it is first necessary to perform a hash calculation on the query domain name in the update field as a keyword to obtain a hash value (called a second hash result).

[0118] It should be noted that the process of performing hash calculation using the query domain name in the update field as a keyword is similar to the process of performing hash calculation in step S103, and will not be repeated here.

[0119] A2: Determine a first index value based on the first p bits of the second hash result, and calculate the number of leading zeros rho(w) of bits other than the first p bits of the second hash result.

[0120] After obtaining the second hash result, it is necessary to extract the first p bits from the second hash result to determine an index value (called the first index value), which is used to locate a specific bucket in an array. Specifically, the first index value j is obtained by performing a bitwise AND operation on the first p bits of the second hash result and a numerical value (m-1) (i.e., j=h(x)[0:p-1]&(m-1)), where m is the total number of buckets, so that the index value j is within the range of the bucket. At the same time, it is also necessary to calculate the number of leading zeros in the remainder of the second hash result (i.e., the part remaining after removing the first p bits), represented by rho(w). The number of leading zeros can reflect the characteristics or importance of the hash value. In this way, the data structure can be efficiently updated to better process and store relevant information.

[0121] A3: Based on the first index value j and the number of leading zeros rho(w), the M[j]th bucket in the superlogarithm array is updated to obtain an updated superlogarithm array.

[0122] Based on the first index value j and the number of leading zeros rho(w), the M[j]th bucket in the hyperlogarithmic array will be updated to generate an updated hyperlogarithmic array. In this process, the characteristics of the current query or data are first evaluated by calculating the number of leading zeros rho(w), and then this value is compared with the current value of the corresponding M[j]th bucket in the hyperlogarithmic array. Specifically, the value of the updated M[j]th bucket will be the maximum value between the current value M[j] and the newly calculated number of leading zeros rho(w). This update method means that the value recorded in each bucket is kept as the maximum number of leading zeros observed, thereby ensuring that the hyperlogarithmic array can effectively reflect the most significant statistical features. This method is of great significance when processing data because it can improve the accuracy and reliability of query data and provide a better basis for subsequent analysis.

[0123] See also Figure 3 , Figure 3 A method flow chart of a counting minimum sketch updating method provided in an embodiment of the present application. Accordingly, using the query domain name in the update field as a keyword to update the counting minimum sketch of the calculation unit corresponding to the update field can be implemented by steps B1-B3:

[0124] B1: Input the query domain name in the update field as a keyword into the mapped hash functions h0′, h1′, ..., h d-1 ′, and obtain d second index values ​​h0′(x), h1′(x), ..., h d-1 ′(x).

[0125] To update the minimum count sketch, first treat the query domain name in the update field as a keyword and input it into the mapped hash function (h0′, h1′, ..., hd-1′) for hash calculation. This will generate d second index values ​​(h0′(x), h1′(x), ..., hd-1′(x)), which are calculated based on the query domain name x.

[0126] It should be noted that these d second index values ​​have a one-to-one correspondence with each column of the two-dimensional array sketch, that is, the second index value h0′(x) corresponds to the first column of the two-dimensional array sketch, the second index value h1′(x) corresponds to the second column of the two-dimensional array sketch, and so on, the second index value hd-1′(x) corresponds to the d-th column of the two-dimensional array sketch.

[0127] B2: Based on the d second index values ​​and the columns of the two-dimensional array sketch corresponding to the d second index values, determine d items to be calculated from the two-dimensional array sketch.

[0128] Use these d second index values ​​and their corresponding columns of the sketch two-dimensional array to determine the d items to be calculated. Each second index value is used to locate the row of the item to be calculated in the sketch, and the corresponding column is used to determine the position of the item to be calculated in the sketch. The purpose of this step is to clarify which specific count items need to be updated.

[0129] Exemplarily, assuming that the second index value h0′(x)=1, and the second index value h0′(x) corresponds to the first column of the two-dimensional array sketch, then based on the second index value h0′(x), it can be determined that the data in the first row and first column of the two-dimensional array sketch is an item to be calculated.

[0130] B3: Add 1 to each of the d items to be calculated in the two-dimensional array sketch to obtain an updated sketch with the minimum count.

[0131] Finally, the values ​​of the d items to be calculated are added by 1 to obtain the updated minimum count sketch. In this way, the minimum count sketch can reflect the latest query data and improve the accuracy of its estimation of query frequency.

[0132] S105: Based on statistical requirements, data statistics are performed in the updated composite data structure.

[0133] “Based on statistical requirements, data statistics are performed in the updated composite data structure” means that in the updated complex data structure, the data therein are analyzed and calculated according to specific statistical requirements.

[0134] In a possible implementation, the statistical requirements include:

[0135] One or more of counting the total number of queries of all source IPs in the DNS log, counting the query volume of the first target source IP in the DNS log, counting the total query volume of the first target query domain name in the DNS log, counting the query cardinality of the second target source IP in the DNS log, and counting the query volume of the query domain name corresponding to the third target source IP in the DNS log;

[0136] Among them, the query domain name corresponding to the third target source IP and the third target source IP are in the same update field.

[0137] It should be noted that the first target source IP refers to the source IP for which query volume statistics are required. The first target query domain name refers to the query domain name for which total query volume statistics are required. The second target source IP refers to the source IP for which query cardinality statistics are required. The third target source IP refers to the source IP for which query volume statistics of its corresponding query domain name are required.

[0138] See also Figure 4 , Figure 4 A method flow chart of a method for counting the total number of queries of all source IPs in a DNS log provided in an embodiment of the present application. Accordingly, in the updated composite data structure, counting the total number of queries of all source IPs in the DNS log can be achieved through steps C1-C3:

[0139] C1: The two-dimensional array sketch of the minimum count sketch of all the calculation units in the updated composite data structure is used as the array to be calculated.

[0140] To count the total number of queries from all source IPs in the DNS log, first, collect and obtain all the calculation units in the composite data structure of the DNS where the DNS log is located. Each calculation unit contains a two-dimensional array sketch of the minimum counting sketch.

[0141] C2: respectively adding the first column numbers of each array to be calculated to obtain the addition result of each array to be calculated.

[0142] For each array to be calculated (corresponding to the sketch of each calculation unit), perform a specific operation: add the data in the first column respectively to obtain a separate summary result for each array to be calculated.

[0143] C3: Add the addition results of each of the arrays to be calculated to obtain the total number of queries from all source IPs.

[0144] The last step is to aggregate all the summed results in step C2 again, that is, to calculate the total number of queries from all source IPs. This is achieved by summing up the summed results of each array to be calculated. This step combines the feedback from different computing units and ultimately derives the total number of queries from all source IPs covered by the composite data structure. This aggregated result provides a comprehensive overview of the entire system or network activity, which is important for analyzing traffic patterns, identifying hotspot IPs, or optimizing resource allocation.

[0145] Through steps C1-C3, the system can effectively calculate the number of queries stored in each computing unit and aggregate these query numbers to calculate the total number of queries. This process helps to statistically analyze and monitor the query status of different source IPs, providing support for further data processing and decision-making.

[0146] See also Figure 5 , Figure 5 A method flow chart of a method for counting query volume of a target source IP in a DNS log provided by an embodiment of the present application. Accordingly, in the updated composite data structure, counting the query volume of the first target source IP in the DNS log can be achieved through steps D1-D3:

[0147] D1: Use the first target source IP as a keyword to perform hash calculation to obtain a third hash result.

[0148] To count the number of queries for the first target source IP in the DNS log, first, the first target source IP is used as a keyword for hash calculation to generate a third hash result. This hash result will be used in subsequent steps to ensure that the IP can be uniquely identified and related data can be quickly located.

[0149] It should be noted that the process of performing hash calculation using the first target source IP as a keyword is similar to the process of performing hash calculation in step S103, and will not be repeated here.

[0150] D2: In the updated composite data structure, the computing unit corresponding to the first target source IP is determined as the first target computing unit based on the third hash result.

[0151] In the updated composite data structure, the computing unit corresponding to the first target source IP is determined based on the third hash result, and this computing unit is called the first target computing unit. In this way, the specific location storing the IP-related information can be effectively found.

[0152] D3: Add the first column numbers of the two-dimensional array sketch of the minimum counting sketch of the first target computing unit to obtain the query volume of the first target source IP.

[0153] Finally, for the two-dimensional array (sketch) in the minimum counting sketch of the first target computing unit, the values ​​of the first column are added to obtain the query volume of the first target source IP. This statistical result can reflect the query frequency of the IP in a certain period of time and provide a basis for subsequent data analysis and decision-making.

[0154] Through steps D1-D3, the system can effectively count the query volume for a specific source IP and obtain information about the query volume of the source IP from the data of the minimum counting sketch. This process helps to analyze the activities of a specific source IP and optimize data management and analysis strategies.

[0155] See also Figure 6 , Figure 6 A method flow chart of a method for counting the total query volume of a target query domain name in a DNS log provided in an embodiment of the present application. Accordingly, in the updated composite data structure, counting the total query volume of the first target query domain name in the DNS log can be achieved by steps E1-E3:

[0156] E1: querying the one or more update fields where the first target query domain name is located, and determining the one or more update fields where the first target query domain name is located as target fields.

[0157] In order to count the total query volume of the first target query domain name in the DNS log, firstly, one or more update fields corresponding to the first target query domain name are queried and identified, and these fields are determined as target fields, so as to subsequently process the specific query domain name.

[0158] E2: Perform hash calculation on the source IP in each of the target fields to obtain one or more fourth hash results, and determine one or more computing units corresponding to the first target query domain name as second target computing units based on the fourth hash results.

[0159] The source IP in each target field is hashed to obtain one or more fourth hash results. Based on these results, one or more computing units (referred to as second target computing units) corresponding to the first target query domain name can be determined.

[0160] It should be noted that the process of performing hash calculation on the source IP in each target field is similar to the process of performing hash calculation in step S103, and will not be repeated here.

[0161] E3: Calculate and add up the frequency estimation values ​​of the minimum counting sketches corresponding to the first target query domain name in each of the second target calculation units, and use the sum of the frequency estimation values ​​as the total query volume of the first target query domain name in the DNS log.

[0162] Calculate the frequency estimation value of the minimum count sketch corresponding to the first target query domain name in each second target calculation unit, and add these frequency estimation values. The result of this addition is the total query volume of the first target query domain name in the DNS log.

[0163] In a possible implementation, the process of calculating the frequency estimation value of the minimum count sketch corresponding to the first target query domain name in each of the second target calculation units includes:

[0164] The first target query domain name is input into the hash function after the count minimum sketch mapping of the second target calculation unit, and a hash calculation is performed to obtain multiple index values. Finally, the minimum value is selected from these index values ​​as the frequency estimation value of the domain name in the second target calculation unit.

[0165] Specifically, the first target query domain name is respectively input into the hash functions h0′, h1′, ..., hd-1′ mapped by the count minimum sketch of the second target computing unit for hash calculation to obtain d third index values ​​h0′(y), h1′(y), ..., hd-1′(y), and the minimum value of the d third index values ​​is used as the frequency estimation value of the count minimum sketch corresponding to the first target query domain name in the second target computing unit. Wherein, y is the first target query domain name.

[0166] Through steps E1-E3, the system can comprehensively consider the data of the first target query domain name and the related source IP, calculate and estimate the total query volume of the first target query domain name in the DNS log, and provide strong support for deeper data analysis and processing.

[0167] See also Figure 7 , Figure 7 A method flow chart of a method for counting the query cardinality of a target source IP in a DNS log provided by an embodiment of the present application. Accordingly, in the updated composite data structure, counting the query cardinality of the second target source IP in the DNS log can be achieved through steps F1-F3:

[0168] F1: Perform hash calculation using the second target source IP as a keyword to obtain a fifth hash result.

[0169] To count the query cardinality of the second target source IP in the DNS log, first, the second target source IP is used as input to perform a hash calculation to obtain a fifth hash result. The IP address can be mapped to one or more specific locations (i.e., calculation units) in the composite data structure through a hash operation.

[0170] It should be noted that the process of performing hash calculation using the second target source IP as a keyword is similar to the process of performing hash calculation in step S103, and will not be repeated here.

[0171] F2: In the updated composite data structure, the computing unit corresponding to the second target source IP is determined as the third target computing unit based on the fifth hash result.

[0172] Based on the fifth hash result, a corresponding computing unit is found in the updated composite data structure and determined as the third target computing unit.

[0173] F3: Calculate the cardinality estimate of the second target source IP in the superlogarithmic counting unit of the third target computing unit as the query cardinality of the two target source IPs.

[0174] Finally, the cardinality estimation value of the second target source IP in the superlogarithmic counting unit in the third target computing unit is calculated. The calculation formula of the cardinality estimation value is: cardinality estimation value E=alpha*m^2*Z.

[0175] Here, alpha is a calibration constant used to adjust the estimated value to suit the needs of a specific application and is usually determined by experiment or theoretical analysis.

[0176] m represents the number of buckets of the superlogarithmic counting unit of the third target computing unit. The superlogarithmic counting unit is a special data structure that can provide efficient query count estimation in a relatively small storage space.

[0177] Z is an average value determined according to the elements in the superlogarithm array. Specifically, it is expressed as: Z = (1 / m)*sum(2^(-M[i])). Wherein, M[i] is the i-th element in the superlogarithm array of the third target computing unit.

[0178] Through steps F1-F3, the system can calculate the query cardinality of the second target source IP, thereby estimating and analyzing the query frequency or other indicators. This helps to understand the activities of different source IPs and provides a data basis to support further data processing and decision-making.

[0179] See also Figure 8 , Figure 8 A method flow chart of a method for counting the query volume of a query domain name corresponding to a target source IP in a DNS log provided by an embodiment of the present application. Accordingly, in the updated composite data structure, counting the query volume of the query domain name corresponding to the third target source IP in the DNS log can be achieved through steps G1-G3:

[0180] G1: Perform hash calculation using the third target source IP as a keyword to obtain a sixth hash result.

[0181] In order to count the query volume of the query domain name corresponding to the third target source IP in the DNS log, first, a hash calculation is performed on the third target source IP to obtain the positioning information in the calculation unit in the composite data structure. The result obtained here is called the sixth hash result, which is used in the subsequent positioning process.

[0182] G2: In the updated composite data structure, the computing unit corresponding to the third target source IP is determined as the fourth target computing unit based on the sixth hash result.

[0183] Based on the sixth hash result, the computing unit corresponding to the third target source IP is searched and determined in the updated composite data structure as the fourth target computing unit.

[0184] G3: Calculate the frequency estimate of the query domain name corresponding to the third target source IP in the counting minimum sketch of the fourth target computing unit as the query volume of the query domain name corresponding to the third target source IP.

[0185] Finally, the frequency estimation value of the query domain name corresponding to the third target source IP in the counting minimum sketch of the fourth target computing unit is calculated as the query volume of the query domain name corresponding to the source IP.

[0186] The specific calculation process includes inputting the query domain name corresponding to the third target source IP into the d hash functions h0′, h1′, ..., hd-1′ mapped by the counting minimum sketch of the fourth target calculation unit for hash calculation, and obtaining d fourth index values ​​h0′(q), h1′(q), ..., hd-1′(q). Then take the minimum value of these d fourth index values ​​as the frequency estimation value of the query domain name corresponding to the third target source IP in the counting minimum sketch of the fourth target calculation unit. Here, q represents the query domain name corresponding to the third target source IP.

[0187] Through steps G1-G3, the system can effectively calculate the query volume of the query domain name corresponding to the third target source IP, and thereby understand the query behavior of different source IPs and the situation of related query domain names. This is helpful for further data analysis and decision making.

[0188] Based on the contents of S101-S105, it can be known that the DNS log is first traversed to identify one or more query records, and the source IP and query domain name of each record are established. Based on this information, an update field is constructed for each query record. Subsequently, the source IP in the update field is used as a keyword for hash calculation to obtain a first hash result, and the calculation unit corresponding to the update field is determined accordingly. The DNS log adopts a composite data structure, which contains multiple calculation units, each of which is composed of a superlogarithmic counting unit and a counting minimum sketch. Then, the query domain name in the update field is used as a keyword to update the relevant calculation units, thereby generating an updated composite data structure. Finally, according to statistical requirements, data statistical operations are performed in the updated composite data structure. The present application reduces the multiple traversals of the entire log content through the use of hash calculations and composite data structures, thereby reducing computing resources and time consumption. At the same time, the composite data structure combines superlogarithmic counting units and counting minimum sketches, can effectively process large amounts of data, and provide accurate statistical results, and is suitable for high-concurrency query environments.

[0189] See also Fig. 9 , Fig. 9 A schematic diagram of a DNS data statistics device provided in an embodiment of the present application. Fig. 9 As shown, the statistical device of the DNS data includes:

[0190] A first determination module 901 is used to traverse a domain name system DNS log and determine one or more query logs in the DNS log;

[0191] The second determination module 902 is used to determine the corresponding source Internet Protocol IP of each query log and the query domain name in each query log, and construct an update field for each query log based on the source IP and the query domain name of each query log; the update field includes the source IP and the query domain name;

[0192] A first calculation module 903, configured to perform a hash calculation using the source IP in the update field as a keyword to obtain a first hash result;

[0193] A third determination module 904 is used to determine the calculation unit corresponding to the update field based on the first hash result; the DNS where the DNS log is located includes a composite data structure, the composite data structure includes a plurality of the calculation units, each of the calculation units includes a superlogarithmic counting unit and a counting minimum sketch;

[0194] An updating module 905, configured to use the query domain name in the update field as a keyword to update the superlogarithmic counting unit and the counting minimum sketch of the calculation unit corresponding to the update field, so as to obtain an updated composite data structure;

[0195] The data statistics module 906 is used to perform data statistics in the updated composite data structure based on statistical requirements.

[0196] In a possible implementation manner, the device further includes:

[0197] Parameter selection module, used to randomly select parameter p;

[0198] A second calculation module is used to calculate the number m of buckets according to the parameter p; m=2^p;

[0199] A creation module is used to create a super logarithmic array with a size of m and a value of 0; the super logarithmic array is represented as M[0]...M[m-1].

[0200] In a possible implementation manner, the device further includes:

[0201] A random determination module, used for randomly determining a width w and a depth d;

[0202] The initialization module is used to initialize a two-dimensional array sketch of size d*w, where the initial value of each element in sketch is 0; the two-dimensional array sketch is represented as sketch[0...d-1]...[0...w-1];

[0203] Random generation module, used to randomly generate d independent hash functions h0, h1, ..., h d-1 ;

[0204] A mapping module is used to map each hash function to the range of [0,w-1] to obtain the mapped hash functions h0′, h1′, ..., h d-1 '; The d hash functions have a one-to-one correspondence with each column of the two-dimensional array sketch.

[0205] In a possible implementation, the updating module 905 specifically includes:

[0206] A third calculation module, configured to perform a hash calculation using the query domain name in the update field as a keyword to obtain a second hash result;

[0207] A first synthesis module, configured to determine a first index value based on the first p bits of the second hash result, and calculate the number of leading zeros rho(w) of the bits other than the first p bits of the second hash result; the first index value j=h(x)[0:p-1]&(m-1); h(x)[0:p-1] is the first p bits of the second hash result; h(x) is the second hash result;

[0208] a superlogarithmic array updating module, configured to update the M[j]th bucket in the superlogarithmic array based on the first index value j and the number of leading zeros rho(w) to obtain an updated superlogarithmic array;

[0209] Among them, the updated M[j]th bucket is max(M[j], rho(w)).

[0210] In a possible implementation, the updating module 905 further includes:

[0211] The fourth calculation module is used to input the query domain name in the update field as a keyword into the mapped hash functions h0′, h1′, ..., h d-1 ′, and obtain d second index values ​​h0′(x), h1′(x), ..., h d-1'(x); x is the query domain name in the update field; d second index values ​​have a one-to-one correspondence with each column of the two-dimensional array sketch;

[0212] a fourth determination module, configured to determine d items to be calculated from the two-dimensional array sketch based on d second index values ​​and the columns of the two-dimensional array sketch corresponding to the d second index values; the second index values ​​are used to determine the rows in the two-dimensional array sketch where the items to be calculated are located, and the columns in the two-dimensional array sketch corresponding to the second index values ​​are used to determine the columns in the two-dimensional array sketch where the items to be calculated are located;

[0213] The counting minimum sketch updating module is used to add 1 to the d items to be calculated in the two-dimensional array sketch respectively to obtain an updated counting minimum sketch.

[0214] In a possible implementation, the statistical requirements include:

[0215] One or more of counting the total number of queries of all source IPs in the DNS log, counting the query volume of the first target source IP in the DNS log, counting the total query volume of the first target query domain name in the DNS log, counting the query cardinality of the second target source IP in the DNS log, and counting the query volume of the query domain name corresponding to the third target source IP in the DNS log;

[0216] Among them, the query domain name corresponding to the third target source IP and the third target source IP are in the same update field.

[0217] In a possible implementation, the data statistics module 906 specifically includes:

[0218] A first setting module is used to use the two-dimensional arrays of the minimum count sketches of all the calculation units in the updated composite data structure as the array to be calculated;

[0219] A fifth calculation module, used for respectively adding the first column numbers of each of the arrays to be calculated to obtain the addition result of each of the arrays to be calculated;

[0220] The sixth calculation module is used to add the addition results of each of the arrays to be calculated to obtain the total number of queries from all source IPs.

[0221] In a possible implementation, the data statistics module 906 further includes:

[0222] A seventh calculation module, configured to perform a hash calculation using the first target source IP as a keyword to obtain a third hash result;

[0223] A second setting module is used to determine, in the updated composite data structure, a computing unit corresponding to the first target source IP as a first target computing unit based on the third hash result;

[0224] An eighth calculation module is used to add the first column numbers of the two-dimensional array sketch of the minimum counting sketch of the first target calculation unit to obtain the query volume of the first target source IP.

[0225] In a possible implementation, the data statistics module 906 further includes:

[0226] a second integration module, configured to query the one or more update fields where the first target query domain name is located, and determine the one or more update fields where the first target query domain name is located as target fields;

[0227] A third integration module is used to perform hash calculation on the source IP in each of the target fields to obtain one or more fourth hash results, and determine one or more computing units corresponding to the first target query domain name as second target computing units based on the fourth hash results;

[0228] a fourth integration module, configured to calculate and add frequency estimation values ​​of the minimum counting sketches corresponding to the first target query domain name in each of the second target calculation units, and use the addition result of the frequency estimation values ​​as the total query volume of the first target query domain name in the DNS log;

[0229] Among them, the calculation process of the frequency estimation value of the counting minimum sketch corresponding to the first target query domain name in each of the second target computing units includes: inputting the first target query domain name into the hash functions h0′, h1′, ..., hd-1′ mapped by the counting minimum sketch of the second target computing unit for hash calculation, obtaining d third index values ​​h0′(y), h1′(y), ..., hd-1′(y), and taking the minimum value of the d third index values ​​as the frequency estimation value of the counting minimum sketch corresponding to the first target query domain name in the second target computing unit; y is the first target query domain name.

[0230] In a possible implementation, the data statistics module 906 further includes:

[0231] A ninth calculation module, configured to perform a hash calculation using the second target source IP as a keyword to obtain a fifth hash result;

[0232] A third setting module, configured to determine, in the updated composite data structure, a computing unit corresponding to the second target source IP as a third target computing unit based on the fifth hash result;

[0233] A fourth setting module, used for calculating the cardinality estimation value of the second target source IP in the superlogarithm counting unit of the third target calculation unit as the query cardinality of the second target source IP;

[0234] Among them, the cardinality estimate E=alpha*m^2*Z; alpha is the calibration constant; m is the number of buckets of the superlogarithmic counting unit of the third target computing unit; Z is the average value, Z=(1 / m)*sum(2^(-M[i])), i=0,...,m-1; M[i] is the element in the superlogarithmic array of the superlogarithmic counting unit of the third target computing unit.

[0235] In a possible implementation, the data statistics module 906 further includes:

[0236] a tenth calculation module, configured to perform a hash calculation using the third target source IP as a keyword to obtain a sixth hash result;

[0237] A fifth determination module, configured to determine, in the updated composite data structure, the computing unit corresponding to the third target source IP as the fourth target computing unit based on the sixth hash result;

[0238] A fifth setting module, configured to calculate a frequency estimation value of the query domain name corresponding to the third target source IP in the counting minimum sketch of the fourth target calculation unit as a query amount of the query domain name corresponding to the third target source IP;

[0239] Among them, the calculation process of the frequency estimation value of the query domain name corresponding to the third target source IP in the counting minimum sketch of the fourth target computing unit includes: inputting the query domain name corresponding to the third target source IP into the hash functions h0′, h1′, ..., hd-1′ mapped by the counting minimum sketch of the fourth target computing unit for hash calculation, obtaining d fourth index values ​​h0′(q), h1′(q), ..., hd-1′(q), and taking the minimum value of the d fourth index values ​​as the frequency estimation value of the query domain name corresponding to the third target source IP in the counting minimum sketch of the fourth target computing unit; q is the query domain name corresponding to the third target source IP.

[0240] This application can significantly reduce the number of DNS log traversals and improve data processing efficiency through hash calculations and composite data structures (hyperlogarithmic counting units and counting minimum sketches). Traditional DNS log analysis requires a lot of computing resources, but this application reduces the consumption of computing resources by optimizing data structures and statistical methods, making it more economical and efficient when processing large amounts of data.

[0241] The above is a detailed introduction to a DNS data statistics method and device provided by the present application. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

[0242] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0243] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A statistical method for DNS data, characterized in that: The method comprises: Traversing a domain name system (DNS) log to determine one or more query logs in the DNS log; Determine the corresponding source Internet Protocol IP of each query log and the query domain name in each query log, and construct an update field for each query log based on the source IP and the query domain name of each query log; the update field includes the source IP and the query domain name; The source IP in the update field is used as a keyword to perform a hash calculation to obtain a first hash result, and a calculation unit corresponding to the update field is determined based on the first hash result; the DNS where the DNS log is located includes a composite data structure, the composite data structure includes a plurality of the calculation units, each of the calculation units includes a superlogarithmic counting unit and a counting minimum sketch; Using the query domain name in the update field as a keyword, updating the superlogarithmic counting unit and the counting minimum sketch of the calculation unit corresponding to the update field, to obtain an updated composite data structure; Based on statistical requirements, data statistics are performed in the updated composite data structure.

2. The method according to claim 1, characterized in that The method further comprises: Initialize the superlogarithmic counting unit: Randomly select a parameter p, and calculate the number of buckets m according to the parameter p; m = 2^p; Create a super logarithmic array of size m and value 0; the super logarithmic array is represented as M[0]...M[m-1]; Initialize the counting minimum sketch: Randomly determine the width w and depth d; Initialize a two-dimensional array sketch of size d*w, where the initial value of each element in sketch is 0; the two-dimensional array sketch is represented by sketch[0...d-1]...[0...w-1]; Randomly generate d independent hash functions h0, h1, ..., h d-1 , and map each hash function to the range of [0,w-1] to obtain the mapped hash functions h0′, h1′, ..., h d-1 '; The d hash functions have a one-to-one correspondence with each column of the two-dimensional array sketch.

3. The method according to claim 2, characterized in that Using the query domain name in the update field as a keyword to update the superlogarithm counting unit of the calculation unit corresponding to the update field includes: Performing a hash calculation using the query domain name in the update field as a keyword to obtain a second hash result; Determine a first index value based on the first p bits of the second hash result, and calculate the number of leading zeros rho(w) of the bits other than the first p bits of the second hash result; the first index value j=h(x)[0:p-1]&(m-1); h(x)[0:p-1] is the first p bits of the second hash result; h(x) is the second hash result; Update the M[j]th bucket in the superlogarithm array based on the first index value j and the number of leading zeros rho(w) to obtain an updated superlogarithm array; Among them, the updated M[j]th bucket is max(M[j], rho(w)).

4. The method according to claim 2, characterized in that: Using the query domain name in the update field as a keyword to update the counting minimum sketch of the computing unit corresponding to the update field includes: The query domain name in the update field is input as a keyword into the mapped hash functions h0′, h1′, ..., h d-1 ′, and obtain d second index values ​​h0′(x), h1′(x), ..., h d-1 '(x); x is the query domain name in the update field; d second index values ​​have a one-to-one correspondence with each column of the two-dimensional array sketch; Based on d second index values ​​and the columns of the two-dimensional array sketch corresponding to the d second index values, d items to be calculated are determined from the two-dimensional array sketch; the second index values ​​are used to determine the rows in the two-dimensional array sketch where the items to be calculated are located, and the columns in the two-dimensional array sketch corresponding to the second index values ​​are used to determine the columns in the two-dimensional array sketch where the items to be calculated are located; The d items to be calculated in the two-dimensional array sketch are respectively increased by 1 to obtain an updated sketch with minimum count.

5. The method according to claim 1, characterized in that The statistical requirements include: One or more of counting the total number of queries of all source IPs in the DNS log, counting the query volume of the first target source IP in the DNS log, counting the total query volume of the first target query domain name in the DNS log, counting the query cardinality of the second target source IP in the DNS log, and counting the query volume of the query domain name corresponding to the third target source IP in the DNS log; Among them, the query domain name corresponding to the third target source IP and the third target source IP are in the same update field.

6. The method according to claim 5, characterized in that In the updated composite data structure, the total number of queries from all source IPs in the DNS log is counted, including: The two-dimensional arrays of the minimum count sketches of all the calculation units in the updated composite data structure are all used as the array to be calculated; Adding the first column numbers of each of the arrays to be calculated respectively to obtain the addition results of each of the arrays to be calculated; The addition results of each of the arrays to be calculated are added together to obtain the total number of queries from all source IPs.

7. The method according to claim 5, characterized in that In the updated composite data structure, the query volume of the first target source IP in the DNS log is counted, including: Performing hash calculation using the first target source IP as a keyword to obtain a third hash result; In the updated composite data structure, determining the computing unit corresponding to the first target source IP as the first target computing unit based on the third hash result; The first column numbers of the two-dimensional array sketch of the minimum counting sketch of the first target computing unit are added to obtain the query volume of the first target source IP.

8. The method according to claim 5, characterized in that In the updated composite data structure, counting the total query volume of the first target query domain name in the DNS log includes: querying the one or more update fields where the first target query domain name is located, and determining the one or more update fields where the first target query domain name is located as target fields; Performing hash calculation on the source IP in each of the target fields to obtain one or more fourth hash results, and determining one or more computing units corresponding to the first target query domain name as second target computing units based on the fourth hash results; Calculate and add the frequency estimation values ​​of the minimum counting sketches corresponding to the first target query domain name in each of the second target calculation units, and use the sum of the frequency estimation values ​​as the total query volume of the first target query domain name in the DNS log; Among them, the calculation process of the frequency estimation value of the counting minimum sketch corresponding to the first target query domain name in each of the second target computing units includes: inputting the first target query domain name into the hash functions h0′, h1′, ..., hd-1′ mapped by the counting minimum sketch of the second target computing unit for hash calculation, obtaining d third index values ​​h0′(y), h1′(y), ..., hd-1′(y), and taking the minimum value of the d third index values ​​as the frequency estimation value of the counting minimum sketch corresponding to the first target query domain name in the second target computing unit; y is the first target query domain name.

9. The method according to claim 5, characterized in that In the updated composite data structure, the query cardinality of the second target source IP in the DNS log is counted, including: Performing hash calculation using the second target source IP as a keyword to obtain a fifth hash result; In the updated composite data structure, determining the computing unit corresponding to the second target source IP as the third target computing unit based on the fifth hash result; Calculate the cardinality estimate of the second target source IP in the superlogarithm counting unit of the third target computing unit as the query cardinality of the second target source IP; Among them, the cardinality estimate E=alpha*m^2*Z; alpha is the calibration constant; m is the number of buckets of the superlogarithmic counting unit of the third target computing unit; Z is the average value, Z=(1 / m)*sum(2^(-M[i])), i=0,...,m-1; M[i] is the element in the superlogarithmic array of the superlogarithmic counting unit of the third target computing unit.

10. The method according to claim 5, characterized in that In the updated composite data structure, counting the query volume of the query domain name corresponding to the third target source IP in the DNS log includes: Performing a hash calculation using the third target source IP as a keyword to obtain a sixth hash result; In the updated composite data structure, determining the computing unit corresponding to the third target source IP as the fourth target computing unit based on the sixth hash result; Calculate the frequency estimation value of the query domain name corresponding to the third target source IP in the counting minimum sketch of the fourth target calculation unit as the query amount of the query domain name corresponding to the third target source IP; Among them, the calculation process of the frequency estimation value of the query domain name corresponding to the third target source IP in the counting minimum sketch of the fourth target computing unit includes: inputting the query domain name corresponding to the third target source IP into the hash functions h0′, h1′, ..., hd-1′ mapped by the counting minimum sketch of the fourth target computing unit for hash calculation, obtaining d fourth index values ​​h0′(q), h1′(q), ..., hd-1′(q), and taking the minimum value of the d fourth index values ​​as the frequency estimation value of the query domain name corresponding to the third target source IP in the counting minimum sketch of the fourth target computing unit; q is the query domain name corresponding to the third target source IP.

11. A statistical device for DNS data, characterized in that: The device comprises: A first determination module is used to traverse a domain name system DNS log and determine one or more query logs in the DNS log; A second determination module is used to determine the corresponding source Internet Protocol IP of each query log and the query domain name in each query log, and to construct an update field for each query log based on the source IP and the query domain name of each query log; the update field includes the source IP and the query domain name; A first calculation module, configured to perform a hash calculation using the source IP in the update field as a keyword to obtain a first hash result; A third determination module is used to determine the calculation unit corresponding to the update field based on the first hash result; the DNS where the DNS log is located includes a composite data structure, the composite data structure includes a plurality of the calculation units, each of the calculation units includes a superlogarithmic counting unit and a counting minimum sketch; An updating module, configured to use the query domain name in the update field as a keyword to update the superlogarithmic counting unit and the counting minimum sketch of the calculation unit corresponding to the update field, so as to obtain an updated composite data structure; The data statistics module is used to perform data statistics in the updated composite data structure based on statistical requirements.

Images