Domain name resolution buffering method for web crawler

Abstract

The invention provides a domain name resolution buffering method for web crawlers, which belongs to the technical field of data processing, and comprises the following steps of: on the basis of deeply analyzing a domain name resolution working mechanism of an operating system and a JVM (Java Virtual Machine), combining the characteristics of parallelization, task multi-node homogenization and the like of distributed crawlers, and adopting a positive and negative double-cache mode to realize the domain name resolution buffering of the web crawlers. A specific data structure and an expiration strategy are respectively designed and adopted for the two caches. Experiments show that the method can effectively improve the domain name resolution performance, reduce the memory occupation of each crawler node and the influence of the domain name resolution request on the network bandwidth, and improve the overall performance of the distributed crawler.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a domain name resolution buffering method for web crawlers. Background technique [0002] Search engines are currently the most efficient way to get information from the Internet. As the basis of search engines, distributed crawlers have been extensively researched and applied. They usually consist of multiple components such as URL analyzers, DNS caches, and rate control. When a crawler crawls a web page, it needs to use DNS (DomainName Service) to convert the domain name of the target host into an IP address. Studies have shown that this link is one of the main performance bottlenecks of crawlers. [0003] At home and abroad, in-depth research has been carried out on the strategy and performance of single-machine crawlers. The research on distributed crawlers mainly focuses on crawler strategies such as task scheduling and resource allocation. Research on the performa...

AI Technical Summary

Problems solved by technology

There are two problems with this data structure. First, the space efficiency is low. Distributed crawlers need to process a large number of domain names. If IVM’s cache data structure is used to store these first-level domain names, only the redundant storage of TLD needs 5×3 ×109=1.5GB; Considering that a website usually has multiple sub-domain names, at this time, TLD redundancy requires more space (assuming that each website has an average of 3 sub-domain names, TLD redundancy requires close to 4.5GB of memory space)

Secondly, the data of this structure is only shared within the same JVM, which does not conform to the application characteristics of distributed crawler multi-node sharing and distributed parallelism, and the simultaneous access of this data structure needs to be blocked by synchronous locks, which cannot well support high-level Concurrency will also affect crawler performance

Images