Content aggregation method based on distributed web crawlers

Abstract

The invention provides a content aggregation method based on distributed web crawlers, which comprises the steps that firstly different crawler platforms are arranged at different devices, a request is sent to a crawling network information source end, and the crawler platforms fabricate crawling rules according to target information required by a user and crawl information in which the target user is interested; the crawled network information is processed, similarity detection is carried out based on a data transmission and conversion method in a real-time database and by being combined with a locality sensitive hashing (LSH) method so as to reduce the redundancy of the information; and the information is classified and sorted by the system according to the category, the heat and keywords and then displayed on user equipment. According to the method provided by the invention, LSH and similarity comparison are carried out according to the data information acquired in an actual network so as to acquire a comparison result. Compared with a comparison result acquired by adopting a traditional mode of whole data duplication checking in the prior art, the content aggregation method is higher in calculation speed and more accurate in similarity comparison.

Description

technical field [0001] The present invention relates to the technical field related to web crawlers, in particular to a content aggregation method based on distributed web crawlers. Background technique [0002] With the continuous development of the Internet, the era of big data is coming, and the value of massive data will be more reflected. Due to the increasing amount of Internet information such as massive streaming video resources and rich web content, it is difficult for specific users to accurately and effectively obtain the network data they need through handheld devices in a limited fragmented time period. However, most of the existing content aggregation technologies use simulations based on the upper-level architecture to prove the superiority of their content aggregation systems, and lack the implementation and application of specific information corresponding to the real network environment and target user groups. [0003] The filter conditions selected by the...

AI Technical Summary

Problems solved by technology

[0004] The present invention implements a content aggregation method based on distributed web crawlers, and aims to solve the problem that web crawler technology in the prior art cannot effectively aggregate and classify customized large-scale network information

Images