DNS log compression method and device

A technology of compression method and compression device, which is applied in the Internet field, can solve problems such as impracticability, and achieve good technical effects

Active Publication Date: 2016-05-04
JIUYUAN QIANCHANG BEIJING TECH SERVICE CO LTD
View PDF2 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Among them, the absence of DNS log information of more than one month makes it impossible to analyze DNS log information with a time span of more than one month.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • DNS log compression method and device
  • DNS log compression method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0044] This technical solution relies on the Hadoop big data storage and computing platform, and relies on 2.6 million domain names, original DNS logs, and original Radius logs. The following are related instructions.

[0045] DNS original log:

[0046] When a user accesses a website through a domain name (www.baidu.com), since the Ethernet transmission process is addressed according to the IP address, the DNS client will first query the DNS server for the IP address corresponding to the domain name. Correspondingly, DNS The server will generate a request log (the parsing result field in the request log is empty, and the parsing time field corresponds to the request time), as follows:

[0047] Source IP|Source Port|Destination IP|Destination Port|ID|Domain Name|Request Type|Analysis Result|Analysis Time|Status Code|Request

[0048] 219.141.159.146|11764|219.141.159.146|53|11616|www.baidu.com|A||20151028141117.176|0|q

[0049] Radius log:

[0050] When the user's terminal is...

Embodiment 2

[0073] Preferably, in step 2), when the request port of the DNS information corresponds to the start and end port of the Radius log is 0, it means that this IP address uses all ports of the external network IP, and directly saves the DNS information.

[0074] And, in step 4), the DNS retention log specifically includes:

[0075] User account, domain name, PV, access time point average, access time point variance.

[0076] Further, it is preferred that step 4) further includes: for a certain user, only one record is reserved through the splicing of domain name access information, that is, through the filtering of repeated information.

[0077] Further, preferably, in step 4), calculating the PV, the mean value of the visit time point, and the variance of the visit time point specifically includes:

[0078] The PV value corresponds to the number of times a user visits a certain website;

[0079] The average value is the sum of the minutes from 00:00 of the day to each user’s v...

Embodiment 3

[0082] In a specific embodiment, the method specifically includes:

[0083] According to the source IP address and request port in DNS matching the authentication information of the IP address in the Radius log, only the DNS request port is within the range of the start and end ports of Radius (when the start and end ports of Radius are both 0, it means The IP address uses all the ports of the external network IP), and the DNS information is valid. Taking user A as an example, user A has 12 requests for DNS logs for the domain name "www.baidu.com" within one day, and the time interval is 1 hour, from 8 am to 5 pm, as follows.

[0084] 219.141.159.146|11764|219.141.159.146|53|11616|www.baidu.com|A||20151028080000.176|0|q

[0085] 219.141.159.146|11764|219.141.159.146|53|11736|www.baidu.com|A||20151028090000.321|0|q

[0086] 219.141.159.146|11764|219.141.159.146|53|13211|www.baidu.com|A||20151028100000.390|0|q

[0087] 219.141.159.146|11764|219.141.159.146|53|17141|www.baidu....

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a DNS log compression method and device. The method comprises: a step 1), obtaining an original log of a domain name system DNS; a step 2), matching authentication information of an IP address in a Radius log according to a source IP address in the DNS and a request port; a step 3), judging whether the domain name of the DNS belongs to 2600 thousands of domain names; and a step 4), when the domain name belongs to the DNS in the 2600 thousands of domain names, analyzing the original log of the DNS, and matching a user account, a computing PV, an access time point mean and an access time point variance, finally merging multiple records into one record at last, and meanwhile deleting the original log. After the DNS log compression method and device provided by the scheme of the invention are adopted, more than 100 orders of magnitudes of the original data of the DNS are reduced, and the value of the data is guaranteed while greatly reducing the data size of the DNS, so that the technical effect is very good.

Description

technical field [0001] The invention belongs to the field of the Internet, and in particular relates to a domain name system log DNS compression method and device. Background technique [0002] With the rapid development of Internet technology, more and more users have begun to access and frequently use the Internet. As an indispensable part of the Internet, the amount of logs generated by DNS has also shown an explosive growth. Taking Guangdong Province as an example, every day The original DNS log reached 12T, extended to the whole country, the daily DNS original log volume reached 300T, for DNS providers, how to spend the minimum cost to store DNS logs and ensure the validity of the data has become an urgent problem to be solved . [0003] method one: [0004] Compress DNS logs and build a storage cluster large enough to store compressed DNS logs. Among them, because the expansion of the cluster will consume more storage, the cost will increase as time goes by. [000...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L29/12H04L29/08H04L12/24G06F17/30
CPCH04L41/069G06F16/1744H04L61/4511H04L67/5651
Inventor 丁文涛尹嘉路
Owner JIUYUAN QIANCHANG BEIJING TECH SERVICE CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap