Content aggregation method based on distributed web crawlers

A technology of distributed network and content aggregation, applied in the field of content aggregation based on distributed web crawler, can solve the problem of network information aggregation and classification that cannot be customized in large quantities, and achieve the effect of fast calculation speed and accurate similarity comparison.

CN105279272AInactive Publication Date: 2016-01-27江苏未来网络集团有限公司
5 Cites 43 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2016-01-27
Estimated Expiration
Not applicable · inactive patent

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention provides a content aggregation method based on distributed web crawlers, which comprises the steps that firstly different crawler platforms are arranged at different devices, a request is sent to a crawling network information source end, and the crawler platforms fabricate crawling rules according to target information required by a user and crawl information in which the target user is interested; the crawled network information is processed, similarity detection is carried out based on a data transmission and conversion method in a real-time database and by being combined with a locality sensitive hashing (LSH) method so as to reduce the redundancy of the information; and the information is classified and sorted by the system according to the category, the heat and keywords and then displayed on user equipment. According to the method provided by the invention, LSH and similarity comparison are carried out according to the data information acquired in an actual network so as to acquire a comparison result. Compared with a comparison result acquired by adopting a traditional mode of whole data duplication checking in the prior art, the content aggregation method is higher in calculation speed and more accurate in similarity comparison.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The present invention relates to the technical field related to web crawlers, in particular to a content aggregation method based on distributed web crawlers. Background technique

[0002] With the continuous development of the Internet, the era of big data is coming, and the value of massive data will be more reflected. Due to the increasing amount of Internet information such as massive streaming video resources and rich web content, it is difficult for specific users to accurately and effectively obtain the network data they need through handheld devices in a limited fragmented time period. However, most of the existing content aggregation technologies use simulations based on the upper-level architecture to prove the superiority of their content aggregation systems, and lack the implementation and application of specific information corresponding to the real network environment and target user groups.

[0003] The filter conditions selected by the...

Examples

Embodiment Construction

[0029] The present invention provides a content aggregation method based on a distributed web crawler. In order to make the object, technical solution and effect of the present invention clearer and clearer, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific implementations described here are only used to explain the present invention, not to limit the present invention.

[0030] The structural diagram of the content aggregation system provided by the present invention is as follows: Figure 6 As shown, the system includes:

[0031] User interface: the user manages the system and schedules tasks through the graphical user interface. The scheduling service is handled by each node crawler, which mainly provides crawler task start, task stop and task status services; the graphical user interface is the visualization provided to the user by the content aggregation platform Operat...