Link deduplication method, device, equipment and storage medium based on web crawler

Abstract

The invention relates to the technical field of the Internet, and discloses a network crawler-based link deduplication method, device, equipment and storage medium. The method includes: when receiving a data capture request of an agricultural product to be analyzed, extracting the first URL link of the platform to be accessed from the data capture request; sending an access request to the platform to be accessed according to the first URL link ; After receiving the response from the platform to be accessed according to the access request, grab the data information in the page corresponding to the first URL link; analyze the data information to obtain the second URL link embedded in the page, and convert the second URL link The URL link is added to the URL queue to be crawled; the counting Bloom filter of the link feature is used, and combined with multiple hashes, the second URL link in the URL queue to be crawled is jointly deduplicated. The present invention improves the performance of the web crawler by optimizing the link deduplication mode, thereby ensuring that the web crawler can quickly obtain the information required by people and improving user experience.

Description

technical field [0001] The present invention relates to the technical field of the Internet, in particular to a web crawler-based link deduplication method, device, equipment and storage medium. Background technique [0002] Web crawlers will inevitably encounter repeated downloads of web pages when crawling web pages. In order to prevent the efficiency of web crawlers from repeatedly crawling and wasting server resources, it is necessary to implement Uniform Resource Locator (Uniform Resource Locator, URL) to filter and deduplicate. At present, common link deduplication methods include: link compression deduplication based on the fifth-generation message digest algorithm (message-digest algorithm 5, MD5), storage deduplication based on hash algorithm, and link deduplication based on Bloom filter Etc. to deduplicate links. [0003] Although, the MD5-based link compression and deduplication method solves the problem that a Uniform Resource Locator (Uniform Resource Locator,...

AI Technical Summary

Problems solved by technology

However, with more and more URLs, the memory space occupancy rate will become higher and higher, and the feature of low conflict probability will reduce the accuracy of duplicate checking, so it will seriously affect the performance of web crawlers

[0004] Although the storage deduplication method based on the hash algorithm is fast and accurate, it needs to design a good hash function and maintain the hash table

In addition, as the scale of crawling web pages increases, the memory consumption will be too high, which will seriously affect the performance of web crawlers

[0005] Although the link deduplication method based on the Bloom filter can solve the problem of space complexity, it has certain misjudgments and cannot delete existing elements.

In other words, the more elements there are, the greater the false positive rate will be, which will seriously affect the performance of the web crawler

Images