Efficient DNS traffic log lossy compression method and device

Abstract

The invention relates to the field of data compression, in particular to efficient compression of DNS traffic logs. In the current mainstream technology, a gzip algorithm is generally adopted to directly compress a DNS flow log file. Due to the fact that the compression effect in the prior art is limited, in order to save storage space more effectively, a compression technology based on timestamp difference, TOP domain name coding, DNS address coding and domain name distribution recombination is adopted to process DNS traffic logs. For domain name compression, common TOP domain name codes are used for replacement. For DNS address compression, full-amount DNS address codes are adopted for replacement, and the data volume is further reduced. For timestamp compression, a current time date is converted into a timestamp, and then the timestamp is replaced by a time difference value. In addition, aiming at the problem that the compression effect of a large-section DNS traffic log is poor, the invention provides a new method: compression is carried out by using a domain name distribution recombination technology. Therefore, by improving the similarity of the DNS traffic logs, the gzip compression effect is further improved.

Description

Technical Field

[0001] The present invention relates to the field of data compression, and in particular to an efficient DNS traffic log lossy compression method and device. Background Art

[0002] Generally, collecting DNS traffic logs involves collecting DNS traffic messages within a certain period of time, and recording the key information in these messages, such as timestamps, domain names, and IP addresses, in a text format after parsing. However, facing a large volume of DNS messages, a large amount of highly redundant text data is often generated, which poses a considerable challenge to storage resources. In order to effectively save storage space, a common practice is to directly use the gzip algorithm to compress the traffic logs.

[0003] Gzip is the abbreviation of GNU zip, which is a GNU free software file compression program and is often used to represent the gzip file format. The authors of the software are Jean-loup Gailly and Mark Adler. The basis of gzip is DEFLATE, which is a combination of LZ77 and Huffman coding. DEFLATE was originally designed as an alternative version of LZW and other patent-protected data compression algorithms. At that time, those patents restricted the application of compress and other popular archiving tools. The basic principle of compressing files is to find repeated bytes in the file, and create a dictionary file with the same bytes and represent them with a code. For example, if there are several identical words in the file, they are represented by a code and written into the dictionary file, so that the purpose of reducing the file can be achieved.

[0004] The current mainstream technology generally uses the gzip algorithm to directly compress DNS traffic log files (see Figure 1 ). However, the advantage of gzip compression technology is that it can significantly improve the compression rate for text data with short length and continuous high redundancy. However, for text files with large volume and non-continuous redundancy distribution, such as large-volume DNS traffic logs, the compression effect of gzip is relatively limited, and it is difficult to achieve the ideal storage space saving goal. Summary of the invention

[0005] In view of the limited compression effect of the prior art, in order to save storage space more effectively, the present invention recommends using a compression technology based on timestamp difference, TOP domain name encoding, DNS address encoding and domain name distribution reorganization to process DNS traffic logs.

[0006] In terms of domain name compression, given that domain names often occupy tens or even hundreds of bytes of storage space, the present invention recommends using commonly used TOP domain name codes for replacement, thereby significantly reducing storage requirements. For the compression of DNS addresses, the present invention recommends using the full amount of DNS address codes for replacement to further reduce the amount of data. In terms of timestamp compression, the present invention first converts the current time and date into a timestamp, and then uses the time difference to replace it to reduce the storage space occupied by the time information. This strategy can significantly reduce the storage requirements of time and date information that originally required more than 14 bytes. In addition, in response to the problem of poor compression effect of large sections of DNS traffic logs, the present invention proposes a new method: using domain name distribution reorganization technology for compression. During the processing process, the present invention will be based on the proportion of the number of lines of the existing DNS traffic log TOP100000 domain names in the system, and then the number of files reorganized as needed, while ensuring that a single domain name is only stored in a unique file, all domain names are allocated to a specific new file. In this way, the present invention can improve the similarity of DNS traffic logs, thereby further improving the effect of gzip compression.

[0007] In summary, this comprehensive compression strategy can utilize storage space more efficiently and meet the storage requirements of large-volume DNS logs.

[0008] 1. The domain name compression process is as follows: First, the TOP 100,000 most commonly used domain names in the current system are collected and encoded in the range of 0 to 99,999. In order to distinguish these encoded domain names from the original domain names or other non-TOP domain names, we add a specific identifier ":" (colon) at the beginning of each encoding. For domain names that are not in the TOP100,000, they remain unchanged without any encoding or adding identifiers. During the decompression process, text containing the identifier ":" is specially processed. It will compare the previously saved TOP100,000 domain name table and restore the encoding with the identifier to the corresponding original domain name. For those texts without identifiers, they are directly kept as they are.

[0009] This domain name compression method may have certain limitations in some systems, especially for those very few but very short and frequently occurring domain names, the compression effect may not be ideal. In addition, in order to support this compression and decompression operation, it is necessary to maintain a comparison table containing up to the TOP 100,000 domain names, which increases the storage and management costs to a certain extent. However, for most systems containing a large number of different domain names, this method is still an effective compression method and can significantly reduce the storage space requirements of domain names.

[0010] 2. The DNS address encoding compression optimization process is as follows: Given that the total number of DNS addresses is relatively limited, we will first systematically collect the DNS addresses that are frequently used in the current environment. Then, these addresses are sorted from high to low according to the frequency of their appearance in the log. On this basis, a unique code starting from 0 is assigned to each DNS address, and this code is used to replace the original DNS address in the log. For a very small number of uncommon DNS addresses, they remain unchanged, thereby achieving compression of log data. During the decompression process, special processing will be performed on texts whose DNS addresses are purely numeric. It will compare the previously saved DNS address encoding table and restore the encoding to the corresponding original DNS address.

[0011] In the decompression process, facing the text that has been replaced by encoding, we will restore the original DNS addresses one by one according to the commonly used DNS address comparison table built in advance. It is worth noting that in order to support this compression and decompression process, the system must maintain a detailed and complete DNS address comparison table to ensure accurate data recovery.

[0012] This DNS address compression method not only effectively reduces the storage space of log data, but also improves data processing efficiency. It is especially suitable for scenarios where the DNS address repetition rate is high and there are strict requirements on storage space.

[0013] 3. The timestamp difference compression process is as follows: Timestamp difference compression technology is a log processing strategy, the core of which is to first retain the timestamp of the first line of the log file as the base time. From the second line of the record, the timestamp of each line is converted to the difference relative to the base time, and the original time and date are replaced by this. This conversion process effectively reduces the storage space occupied by timestamp data.

[0014] During the decompression phase, the process restores the original timestamps accurately by reapplying these time differences to the base timestamps. These restored timestamps are then converted back to the standard date and time format to restore the original log time information.

[0015] It is worth noting that this timestamp difference compression method is not very effective for a single log file with a small amount of logs, because the compression effect depends largely on the number of log lines.

[0016] A timestamp is the number of seconds since January 1, 1970 (00:00:00 GMT), also known as a Unix timestamp. Unix timestamp, also known as Unix time or POSIX time, is a time representation method defined as the total number of seconds from 00:00:00 GMT, January 1, 1970 to the present.

[0017] 4. The process of domain name distribution reorganization involves the following steps: First, count the proportion of TOP100000 domain names in the current system. Then, according to the approximate proportion of the number of log lines of these domain names, distribute them to multiple files. In this process, ensure that all log entries of the same domain name are written to only one single file. For domain names with a small proportion of log lines or domain names outside the TOP100000, in order to optimize storage efficiency, the logs of these domain names are merged and written into the same file.

[0018] This strategy ensures that log entries for the same domain are stored together, which significantly improves the redundancy of individual log text. This change greatly enhances the effectiveness of the gzip compression algorithm, because gzip can achieve a higher compression ratio when processing highly repetitive text data. Therefore, the entire reorganization process improves the compression effect and further saves storage space by optimizing the data storage structure.

[0019] It is worth noting that this reorganization strategy will lose the original log order during decompression and cannot perfectly restore the original log. At the same time, it is necessary to collect the proportion information of TOP100000 domain names in advance as the basis for file reorganization.

[0020] Technical advantages For the DNS traffic logs in the same system, the TOP100000 encoding method is used for the domain name, the DNS address encoding method is used for the DNS address, the timestamp difference calculation and the domain name distribution reorganization strategy are used, which saves 15-50% of the storage space compared with the original direct recording and traditional compression methods.

[0021] The DNS domain name TOP100000 encoding and DNS address encoding method proposed in the present invention, wherein the TOP100000 domain name and DNS address encoding are proprietary to the DNS system used and are difficult to completely restore after being obtained by a third party, and therefore can play a role in data encryption and log protection to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is the existing compression process for DNS traffic logs.

[0023] Figure 2 It is the lossy compression process of DNS traffic logs. DETAILED DESCRIPTION

[0024] 1. Establish the TOP100000 domain name coding table Get the TOP100000 domain names according to the number of log requests in the system, and number each domain name 0, 1, 2, 3, ... in sequence. Replace the TOP100000 domain names that appear in the log with numbers, including identifiers, so that they can be represented with a maximum of 6 bytes. The original method of using strings to fully represent domain names generally requires dozens to hundreds of bytes. The TOP100000 domain names account for a large proportion of user request domain names, which can save a lot of space.

[0025] For example: if domain name 1 is encoded as 0 in the TOP100000 file, then replace domain name 1 in the log with ":0", domain name 2 is encoded as 1 in the TOP100000 file, then replace domain name 2 in the log with ":1"... domain name 99999 is encoded as 99999 in the TOP100000 file, then replace domain name 3 in the log with ":99999" Generate some code for the new domain name field: if domain in domain_top_num.keys(): domain_string = “:” + str(domain_top_num[domain]) else: domain_string = domain The number of n in the TOP n domain name needs to be determined according to the actual situation. If n is too large, more bytes are needed to represent the value of the number. If n is too small, some domain names will still occupy more bytes because they cannot be mapped or replaced. The total number of resolutions of the TOP100000 domain names selected in this article can reach more than 95% of the total number of resolutions of all domain names in the log, and the number only needs a maximum of 6 bytes, which can save a lot of storage space.

[0026] 2. Establish DNS address encoding table Obtain the full DNS address according to the ranking of the number of existing log requests in the system, and number each domain name in sequence 0, 1, 2, 3, .... Use the number to replace the DNS address that appears in the log. Generally, the number of addresses does not exceed 100,000, so it can be represented by a maximum of 5 bytes. The original method of using a string to fully represent the DNS address generally requires 10 bytes for IPv4 and more than 10 to 20 bytes for IPv6. The basic repetition of user-requested DNS addresses is very high, and a lot of space can be saved after replacement.

[0027] For example: DNS address 1 is encoded as 0 in the DNS address encoding file, then replace domain name 1 in the log with "0", domain name 2 is encoded as 1 in the DNS address encoding file, then replace domain name 2 in the log with "1"... domain name n is encoded as n in the DNS address encoding file, then replace domain name 3 in the log with "n" Generate some code for the new domain name field: if dns_ip in dns_ip_num: dns_string = str(dns_ip_num[dns_ip]) else: dns_string = dns_ip In the same system, the total resolution times according to the existing log DNS address codes can generally reach more than 99% of the total resolution times of all DNS addresses in the log, and the numbering generally only requires a maximum of 5 bytes, which can save a large amount of storage space.

[0028] 3. Timestamp conversion and time difference calculation The design of this part is to convert the string in time and date format into a timestamp. The time of the first row is used as the starting time, and the difference between the time of each row and the timestamp of the first row is calculated.

[0029] For example, the time and date string of the first line of the log is 20241021094909, which is 09:49:09 on October 21, 2024. It is converted to the timestamp 1729475349 and replaces the time and date string of the first line. The time and date string of the second line of the log is 20241021094910, which is 09:49:10 on October 21, 2024. It is converted to the timestamp 1729475350, which is subtracted from the timestamp 1729475349 of the first line to obtain the difference 1, which replaces the time and date string of the second line... The time and date string of the nth line of the log is Sn, which is subtracted from the timestamp 1729475349 of the first line to obtain the difference Dn, which replaces the time and date string of the nth line.

[0030] Generate some code for new time fields: ts = time.strptime(daytime, "%Y%m%d%H%M%S") line_count[write_file_id] += 1 if line_count[write_file_id] == 1: first_ts = ts timestring = str(ts) else: delta_ts = ts - first_ts timestring = str(delta_ts) The time date format occupies 14 bytes to record the year, month, day, hour, minute, and second. After being converted to a timestamp, it will occupy 10 bytes. The timestamp is the total number of seconds from 00:00:00 on January 1, 1970 to the present. In most cases, it will occupy a fixed 10 bytes. The first line is fixedly reduced by 4 bytes. Usually, the time span of the log will not exceed 1 day, that is, 86400 seconds. Starting from the second line, it occupies 1-5 bytes, which is 9-13 bytes less than the original time date format, about 2 / 3 of the length.

[0031] 4. Distribute by domain name and reorganize and write to multiple files The design of this part is to first allocate the TOP10000 domain names and other domain names into fixed files according to the proportion of the number of times, and write each line of domain names into the corresponding file according to the allocated file.

[0032] For example: if the domain name is loaded into 10 files, each file accounts for about 10%, the TOP1-18 domain names account for about 10% and are written to file 1, the TOP19-77 domain names account for about 10% and are written to file 2, the TOP78-142 domain names account for about 10% and are written to file 3, the TOP143-211 domain names account for about 10% and are written to file 4, the TOP212-310 domain names account for about 10% and are written to file 5, the TOP311-492 domain names account for about 10% and are written to file 6, the TOP493-746 domain names account for about 10% and are written to file 7, the TOP747-1299 domain names account for about 10% and are written to file 8, the TOP1300-2666 domain names account for about 10% and are written to file 9, and the domain names other than TOP2667-100000 and TOP100000 account for about 10% and are written to file 10.

[0033] Generate the code for writing the file number of the load: if domain in top_domain_file_id: file_id = top_domain_file_id[domain] else: file_id = file_id_list[file_count-1] Because the DNS traffic logs of the same domain name have high similarity, when gzip compression is used, the redundancy of strings within a certain size is higher, and the compression effect is better. Depending on the number of payload files, the compression effect can be improved by up to about 50%. It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0034] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0035] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0036] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An efficient lossy compression method for DNS traffic logs, characterized in that: The following steps are involved: (1.1) Based on the number of domain name requests in the DNS traffic log, a TOP domain name encoding table is established, and the TOP domain names appearing in the traffic log are replaced with codes with specific identifiers; (1.2) Establish a DNS address encoding table based on the frequency of occurrence of DNS addresses in DNS traffic logs, and replace the DNS addresses in the traffic logs with the corresponding encodings; (1.3) Convert the time and date fields recorded in the traffic log to Unix timestamps, and use the timestamp of the first record as the benchmark, and replace the original time and date fields by calculating the time difference between subsequent records and the benchmark time; (1.4) According to actual business needs, the traffic log entries are evenly reorganized and distributed into N files to ensure that the log entries of each domain name are stored in only a single file; (1.5) Perform gzip compression on the traffic log files processed by (1.4) in turn.

2. The method according to claim 1, characterized in that The TOP domain name encoding table is generated as follows: (2.1) Sort the domain names in the traffic log by the number of requests from high to low, and select the top 100,000 domain names as the TOP domain names; (2.2) Assign a unique number to each TOP domain name, ranging from 0 to 99,999; (2.3) Add a colon identifier ":" before the encoding to distinguish the encoded TOP domain name from the non-encoded domain name.

3. The method according to claim 1, characterized in that The DNS address encoding table is generated as follows: (3.1) Sort the DNS addresses in the traffic log by request frequency from high to low; (3.2) Sort by DNS address request frequency and assign a unique code starting from 0 to each DNS address; (3.3) The DNS address fields appearing in the traffic logs are replaced with corresponding number codes, and the unencoded DNS addresses are kept in their original form.

4. The method according to claim 1, characterized in that The time difference is calculated as follows: (4.1) Convert the time and date format string in the traffic log to a Unix timestamp; (4.2) Taking the timestamp of the first record in the log as the base time, calculate the time difference of the timestamp of each subsequent log record relative to the base time; (4.3) Replace the time and date fields in the original log record with the calculated time difference.

5. The method according to claim 1, characterized in that Traffic log entry reassembly includes the following specific steps: In descending order of the number of domain name requests, the traffic log entries corresponding to the domain names accounting for 1 / N of the total entries are written into N files.

6. An efficient DNS traffic log lossy compression device, characterized in that: Includes the following modules: (6.1) TOP domain name encoding module, used to generate the TOP domain name encoding table and replace the TOP domain name field in the log with the code with the identifier; (6.2) DNS address encoding module, used to generate a DNS address encoding table and replace the DNS address field in the log with the corresponding encoding; (6.3) Timestamp compression module, used to convert the time date field into Unix timestamp and calculate the time difference to replace the original time field; (6.4) Log reorganization module, which redistributes traffic log entries into multiple files and ensures that the number of entries in each file is equal; (6.5) gzip compression module, used to apply the gzip compression algorithm to the reorganized log files to save storage space.

7. The device according to claim 6, characterized in that: The encoding coverage of TOP domain names reaches over 95%, and the encoding coverage of DNS addresses reaches over 99% to achieve efficient compression effects.

8. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, characterized in that: When the instructions are executed by a computing device, the computing device is caused to perform the method according to any one of claims 1 to 5.

9. An electronic device, characterized in that: The method comprises one or more processors, one or more memories and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method as claimed in any one of claims 1 to 5.

Images