TOP N hot domain name real-time monitoring method based on HeavyKeeper algorithm

Abstract

This invention relates to a real-time monitoring method for TOP N hot domain names based on the HeavyKeeper algorithm. This method periodically processes requested domain names received in real-time by DNS, specifically including: calculating the hash value Hdomain of the current requested domain name; querying whether a corresponding node for Hdomain exists in the Red-Black Tree (RB Tree), where the RB Tree contains the same N nodes as the MinHeap; if the query finds a match, updating the value Dnum of the corresponding node in the RB Tree and MinHeap, where Dnum is an approximation of the number of times the requested domain name has been accessed; and at the end of the current period, outputting the TOP N hot domain names. Compared with existing technologies, this invention has significant advantages: it uses configurable, fixed-quantity memory, only records the access counts of potentially hot domain names, eliminates domain names with low access counts without storing them, avoiding wasted memory resources; and when multiple domain name hash storage conflicts are detected, probability decay is used to remove domain names with low access counts, improving the accuracy of real-time monitoring of TOP N hot domain names.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network data monitoring, and particularly relates to a TOP N hotspot domain name real-time monitoring method based on a HeavyKeeper algorithm. BACKGROUND

[0002] In the process of using a DNS (Domain Name Server) server, it is a common requirement to report the TOP N hotspot domain names appearing in the last X minutes in real time. It is difficult to accurately and timely monitor the first N hotspot domain names (TOP N) using lower computing and memory resources in a high concurrency scenario.

[0003] The existing general implementation methods for this requirement can be divided into two types: the first type is to implement the method in the DNS server software, store the requested domain names in the form of red-black tree nodes, and count the number of times of occurrence. At the Xth moment, the first N domain names with the most counts are output. This method can output the results in time and accurately when the ground concurrency, computing and memory resources are sufficient. The second type is to analyze the DNS server generated log in real time, introduce an additional device to count and sort the access times in X minutes. This method does not consume the resources of the DNS server itself, and can also output the data in time and accurately. The defect of this method is that it needs to invest additional hardware devices.

[0004] The steps of detecting the TOP N hotspot domain names by the two methods can be described as follows:

[0005] 1. Extract the requested domain name Domain, calculate the Hash value of the domain name as Hdomain using the Hash function H, and the value will be used as the Key of the red-black tree node where the domain name is located, which is the unique identifier of the node.

[0006] 2. Query whether the existing red-black tree has inserted the domain name, that is, whether there is a tree node Key = Hdomain. If not, go to the next step, otherwise go to step 4.

[0007] 3. Assign a new tree node to the current domain name, set the tree node Key = Hdomain, and insert the node into the red-black tree. Sometimes, in order to control the total memory occupancy, only M tree nodes, such as 100,000, are retained. When the number exceeds the limit, the domain name data is discarded directly, or some data with a count of 1 is eliminated to release part of the memory space. If the current domain name is discarded, the process is ended. Otherwise, go to the next step.

[0008] 4. Increase the count of the number of times of occurrence of the domain name in the node by 1.

[0009] 5. When the time interval reaches X, such as every 5 minutes, start traversing the red-black tree, sort it according to the number of visits, and output the top N items, which is the result. At the same time, clear the red-black tree to prepare for the next count.

[0010] Existing solutions, in order to monitor the top N hottest domains in real time, must maintain a red-black tree with a considerable number of nodes to count the number of domain accesses. One drawback is that, to improve the accuracy of the statistics while controlling memory usage, a smaller number of nodes (M) results in more data being discarded. Therefore, to improve accuracy, the value of M is often very large, consuming significant computational and memory resources. Another drawback is that, at time X, to obtain the top N values, the entire red-black tree must be traversed to retrieve the N hottest domains with the highest access counts. After outputting, the tree needs to be cleared. Under high concurrency, this operation may cause system instability and packet loss. Therefore, the existing method is generally not suitable for high-concurrency scenarios. Method two is suitable for high-concurrency scenarios, but requires additional equipment investment.

[0011] As can be seen from the implementation steps, both methods have drawbacks in completing the above tasks. Without controlling memory usage, the results are accurate but consume excessive resources. In scenarios where total memory usage is controlled, inaccurate monitoring results may occur due to the discarding of some data.

[0012] Reference 1: Chinese authorized invention patent CN201710736503.9 discloses a method for classifying DNS access data and increasing the proportion of intranet access. Its main function is to classify the intranet and extranet attributes of DNS access data, and realize the simultaneous segmented statistical analysis of classified data and extranet domain name access volume. Among them, the method also saves expired data in real time in the database, which requires a large amount of memory support, and still cannot solve the problem of excessive memory resource consumption. Summary of the Invention

[0013] The purpose of this invention is to provide a real-time monitoring method for TOP N hot domain names based on the HeavyKeeper algorithm. It adopts a dual-process data structure to attenuate and eliminate domain names that appear in small numbers in the array, and only retain domain names with high access frequency, thereby achieving effective monitoring of the accuracy of TOP N hot domain names.

[0014] The technical solution to achieve the purpose of this invention is as follows:

[0015] A real-time monitoring method for TOP N hot domain names based on the HeavyKeeper algorithm is proposed. This method periodically processes the requested domain names received by DNS in real time. The specific processing steps for any given period are as follows:

[0016] Calculate the hash value Hdomain of the current requested domain name, and query whether there is a node with the hash value Hdomain of the corresponding requested domain name in the RB Tree, where the RB Tree contains the same N nodes as MinHeap;

[0017] When a node corresponding to the Hdomain exists in the RB Tree, the requested domain name is in the MinHeap. At the same time, the corresponding node value in the RB Tree and MinHeap is updated to Dnum, where Dnum is the access count value of the current requested domain name.

[0018] When there is no corresponding Hdomain node in the RB Tree, if the number of nodes in the MinHeap exceeds N and the count Nmin of its root node satisfies Dnum-Nmin=1, then the requested domain name is inserted into both the RB Tree and the MinHeap to replace the smallest node; otherwise, if the number of nodes in the MinHeap does not exceed N and the count Nmin of its root node satisfies Dnum-Nmin=1, then the requested domain name is inserted into both the MinHeap and the RB Tree for access accumulation.

[0019] At the end of the current period, output the top N hot domain names.

[0020] Furthermore, if there is no corresponding Hdomain node in the RB Tree, and the count Nmin of the MinHeap root node does not satisfy Dnum-Nmin=1, then the requested domain name is eliminated.

[0021] Furthermore, the requested domain names received by DNS in real time are stored in the memory pool. After outputting the top N hot domain names, the memory pool is cleared.

[0022] Furthermore, the calculation process for Dnum is as follows:

[0023] Initialize an array A with r rows and w columns. Each element in array A includes two fields, which store the hash value of the requested domain name and the number of accesses, respectively.

[0024] Each request domain name is recorded with a unique fingerprint identifier FP. FP is calculated using a hash function and is used to represent the bucket position occupied by each request domain name in array A.

[0025] Perform row traversal on array A;

[0026] Step 1: Set i = 0 to start obtaining the requested domain name;

[0027] Step 2: Calculate the FP corresponding to each request domain name, and obtain the element of the i-th row and [Hi(D)%w] column of each request domain name in array A, i.e., A[i][Hi(D)%w], where Hi(D) represents the hash value calculated by the hash function Hi(·) used in the i-th row for domain name D, and % is the modulo symbol;

[0028] Step 3: Determine the access count for each requested domain name stored in array A. If the access count C = 0, set C to 1; if the access count C > 0 and FP = Hdomain, set C to C+1; if the access count C > 0 and FP ≠ Hdomain, set C to 1.1. -c The probability decay will set C to C-1;

[0029] Step 4: Set Max C = MAX(Max C, C);

[0030] Step 5: If i < r, return to step 3; otherwise, Dnum = Max C.

[0031] Furthermore, in step 3, if the number of visits C > 0 and FP ≠ Hdomain, then 1.1 -c The probability decay is set to C-1. When multiple request domains conflict in the same bucket, the probability decays continuously until the second request domain replaces the first request domain in the current bucket.

[0032] The present invention also provides a TOP N hot domain name real-time monitoring device, including a memory for storing a computer program, and a processor for implementing the steps of the above-described TOP N hot domain name real-time monitoring method based on the HeavyKeeper algorithm when executing the computer program.

[0033] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for real-time monitoring of TOP N hot domain names based on the HeavyKeeper algorithm.

[0034] The significant advantages of this invention compared to existing technologies are:

[0035] 1. By adopting configurable and quantitative memory, the system records only the occurrence frequency of potentially popular domain names, while eliminating domain names with low occurrence frequency and not storing them. This avoids the monitoring data from increasing with the number of domain names and requests, thus solving the problem of excessive memory resource consumption in traditional systems.

[0036] 2. By using a dual-process data structure in conjunction, the consumption of computing resources is evenly distributed. When running inside the business server, in the face of multiple domain name hash storage conflicts, probability decay is used to remove domain names that appear less frequently, eliminating the risk of business system jitter under high concurrency, thereby improving the accuracy of real-time monitoring of TOP N hot domain names. Attached Figure Description

[0037] Figure 1 This is the main flowchart of the TOP N hot domain name real-time monitoring method based on the HeavyKeeper algorithm of the present invention.

[0038] Figure 2 This is a sub-flowchart of the real-time monitoring method for TOP N hot domain names based on the HeavyKeeper algorithm of the present invention. Detailed Implementation

[0039] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0040] like Figure 1 As shown, a real-time monitoring method for TOP N hot domain names based on the HeavyKeeper algorithm is presented. This method periodically processes the requested domain names received by DNS in real time. The specific processing steps for any given period are as follows:

[0041] Set the number of hot domains to be counted, N, and the count interval, Interval;

[0042] The DNS received the request for domain name D;

[0043] Determine whether the current DNS reception time Tnow has reached the statistical time interval Interval. If "no", the DNS continues to receive requested domain names and stores the received requested domain names in a min-heap containing N nodes MinHeap; otherwise, at the end of the current period, output the TOP N hot domain names.

[0044] The DNS stores the requested domain names received in real time into the memory pool (Pool). After outputting the top N hot domain names, the memory pool is cleared.

[0045] The MinHeap's structural feature is that its root node stores the domain name with the fewest occurrences, i.e., Nmin; its time complexity is O(1). When a new domain name enters the MinHeap, it needs to be compared with the Nmin value. If it is greater than the Nmin value, the new domain name is replaced and added to the hot domain name list.

[0046] To determine if a requested domain name is in the MinHeap, the entire MinHeap needs to be traversed, which has a time complexity of O(N). Therefore, the MinHeap is not suitable for directly querying whether a requested domain name is already in the MinHeap. Thus, a Red-Black Tree (RBTree) is introduced. After its introduction, the query time complexity becomes O(log₂N), and the number of domain name nodes stored in the RBTree is the same as that in the MinHeap.

[0047] Specifically, within the current period, calculate the hash value Hdomain of the requested domain name, and check if a node with the corresponding hash value Hdomain exists in the RB Tree. If it exists, the count is Flag=1; otherwise, the count is Flag=0.

[0048] When a node with the corresponding Hdomain exists in the RB Tree, the requested domain name is in the MinHeap. At the same time, the corresponding node value in both the RB Tree and the MinHeap is updated to Dnum, where Dnum is the number of times the current requested domain name has been accessed.

[0049] When there is no corresponding Hdomain node in the RB Tree, if the number of nodes in the MinHeap exceeds N and the count Nmin of its root node satisfies Dnum-Nmin=1, then the requested domain name is inserted into both the RB Tree and the MinHeap to replace the smallest node; otherwise, if the number of nodes in the MinHeap does not exceed N and the count Nmin of its root node satisfies Dnum-Nmin=1, then the requested domain name is inserted into both the MinHeap and the RB Tree for access accumulation.

[0050] If there is no corresponding Hdomain node in the RB Tree, and the count Nmin of the MinHeap root node does not satisfy Dnum-Nmin=1, then the requested domain name is eliminated.

[0051] like Figure 2 As shown, given the input array A, the request domain name, and the Flag value, the calculation process of Dnum is as follows:

[0052] Array A stores a counter for the domain names. It is stored in a fixed-size array of r rows and w columns. Because its memory usage has errors, it needs to be updated in order to accurately record the number of times each requested domain name appears.

[0053] Initialize array A with r rows and w columns, and iterate through array A row by row; set i++ to automatically process the next row of array A after each traversal;

[0054] Step 1: Set i = 0, and enter the requested domain name;

[0055] Specifically, each element in array A includes two fields, which store the hash value of the requested domain name and the number of accesses, respectively; each requested domain name has a unique fingerprint identifier FP, which is calculated using a hash function and occupies 2 bytes, used to represent the bucket position of each requested domain name in array A;

[0056] Step 2: Calculate the FP corresponding to each request domain name, and obtain the element of the i-th row and [Hi(D)%w] column of each request domain name in array A, i.e., A[i][Hi(D)%w], where Hi(D) represents the hash value calculated by the hash function Hi(·) used in the i-th row for domain name D, and % is the modulo symbol;

[0057] Step 3: Determine the access count for each requested domain name stored in array A. The specific update strategy is as follows:

[0058] If the number of visits C = 0, then set C to 1;

[0059] If the number of visits C > 0 and FP = Hdomain, then set C to C+1;

[0060] If the number of visits C > 0 and FP ≠ Hdomain, then C will be set to C-1 with a probability of 1.1-c.

[0061] Step 4: Set Max C = MAX(Max C, C);

[0062] Step 5: If i < r, return to step 3; otherwise, Dnum = Max C.

[0063] Specifically, in step 3, if the number of accesses C > 0 and FP ≠ Hdomain, then C is set to C-1 with a probability decay of 1.1-c. When multiple request domains conflict in the same bucket, the probability of the first request domain being occupied by the second request domain in the current bucket continues to decay until the second request domain replaces the first request domain in occupying the current bucket.

[0064] For example: Configure array A with r = 4 and w = 400,000;

[0065] 1. Enter the requested domain name qqqq.com;

[0066] 2. Calculate FP = HASHFP(qqqq.com), assuming the result is 8000;

[0067] 3. Calculate the four hash values ​​for row r respectively, including: h1 = hash1(qqqq.com) = 1200, h2 = hash2(qqqq.com) = 3600, h3 = hash3(qqqq.com) = 2000, h4(qqqq.com) = 6000;

[0068] 4. Update A[0]

[1200] .C, A[1]

[3600] .C, A[2]

[2000] .C and A[3]

[6000] .C respectively;

[0069] 5. If the domain name pppp.com was requested before the domain name qqqq.com was requested, and the h1 of the domain name pppp.com is 1200, then A[0]

[1200] .FP = HASHFP(pppp.com) = 1111, and A[0]

[1200] .C is set to 1.

[0070] When the requested domain name qqqq.com appears, A[0]

[1200] .FP is identified as 1111, which is inconsistent with the FP=8000 of the requested domain name qqqq.com. Therefore, the requested domain name pppp.com is executed with a probability of 1 / 1.1-1=0.909.

[0071] When the requested domain name qqqq.com appears too many times, as the probability of its appearance decreases, the requested domain name qqqq.com will replace the requested domain name pppp.com. The bucket will be occupied by the requested domain name qqqq.com and set as the FP of the requested domain name qqqq.com.

[0072] Subsequently, the requested domain name bbbb.com and the requested domain name qqqq.com encountered a bucket conflict. The number of accesses to the requested domain name qqqq.com was queried, and the maximum value among A[0]

[1200] .C, A[1]

[3600] .C, A[2]

[2000] .C and A[3]

[6000] .C was taken, and A[0]

[1200] .FP = 8000.

[0073] If the number of visits is 20, the probability of decay will become 1 / 1.1-20=0.15, which is a relatively small probability value. Therefore, the domain name qqqq.com is not likely to be eliminated.

[0074] The present invention also provides a TOP N hot domain name real-time monitoring device, including a memory for storing a computer program, and a processor for implementing the steps of the above-described TOP N hot domain name real-time monitoring method based on the HeavyKeeper algorithm when executing the computer program.

[0075] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the aforementioned real-time monitoring method for TOP N hot domain names based on the HeavyKeeper algorithm. The computer-readable storage medium may include various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0076] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0077] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for real-time monitoring of TOP N hot domain names based on the HeavyKeeper algorithm, characterized in that: This method periodically processes the requested domain names received by DNS in real time. The specific processing steps for any given period are as follows: Calculate the hash value Hdomain of the current requested domain name, and query whether there is a node in the RB Tree that corresponds to the hash value Hdomain of the requested domain name, wherein the RB Tree contains the same N nodes as MinHeap; When a node corresponding to the Hdomain exists in the RB Tree, the requested domain name is in the MinHeap, and the corresponding node value Dnum in the RB Tree and MinHeap is updated simultaneously, where Dnum is the current access count value of the requested domain name. When there is no node corresponding to the Hdomain in the RB Tree, if the number of nodes in the MinHeap exceeds N and the count Nmin of its root node satisfies Dnum-Nmin=1, then the requested domain name is simultaneously inserted into the RB Tree and the MinHeap to replace the smallest node; otherwise, if the number of nodes in the MinHeap does not exceed N and the count Nmin of its root node satisfies Dnum-Nmin=1, then the requested domain name is simultaneously inserted into the MinHeap and the RB Tree for access accumulation. At the end of the current period, output the top N hot domain names.

2. The method for real-time monitoring of TOP N hot domain names based on the HeavyKeeper algorithm according to claim 1, characterized in that: If there is no node corresponding to the Hdomain in the RB Tree, and the count Nmin of the MinHeap root node does not satisfy Dnum-Nmin=1, then the requested domain name is eliminated.

3. The method for real-time monitoring of TOP N hot domain names based on the HeavyKeeper algorithm according to claim 2, characterized in that: The DNS receives request domain names in real time and stores them in the memory pool. After outputting the top N hot domain names, the memory pool is cleared.

4. The method for real-time monitoring of TOP N hot domain names based on the HeavyKeeper algorithm according to claim 3, characterized in that: The calculation process for Dnum is as follows: Initialize an array A with r rows and w columns. Each element in array A includes two fields, which store the hash value of the requested domain name and the number of accesses, respectively. Each requested domain name is recorded with a unique fingerprint identifier FP, which is calculated using a hash function and is used to represent the bucket position occupied by each requested domain name in array A. Perform row traversal on the array A; Step 1: Set i=0 to begin obtaining the requested domain name; Step 2: Calculate the FP corresponding to each of the requested domain names, and obtain the element of the i-th row and [Hi(D)%w] column of each of the requested domain names in the array A, i.e. A[i][Hi(D)%w], where Hi(D) represents the hash value calculated by the domain name D using the hash function Hi(·) used in the i-th row, and % is the modulo symbol; Step 3: Determine the access count of each requested domain name stored in array A. If the access count C = 0, set C to 1; if the access count C > 0 and FP = Hdomain, set C to C+1; if the access count C > 0 and FP ≠ Hdomain, set C to 1.

1. -c The probability decay will set C to C-1; Step 4: Set Max C = MAX(Max C, C); Step 5: If i < r, return to step 3; otherwise, Dnum = Max C.

5. The method for real-time monitoring of TOP N hot domain names based on the HeavyKeeper algorithm according to claim 4, characterized in that: In step 3, if the number of visits C > 0 and FP ≠ Hdomain, then 1.1 -c The probability decay is set to C-1. When multiple request domain names conflict in the same bucket, the probability decay of the number of times the first request domain name currently occupied by the bucket is occupied by the second request domain name continues until the second request domain name replaces the first request domain name in occupying the current bucket.

6. A real-time monitoring device for TOP N hot domain names, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the TOP N hot domain name real-time monitoring method based on the HeavyKeeper algorithm as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the TOP N hot domain name real-time monitoring method based on the HeavyKeeper algorithm as described in any one of claims 1 to 5.

Images