Volunteer computing based multi-tenant professional cloud crawler

Abstract

The present invention relates to the field of network information acquisition, and provides a volunteer computing based multi-tenant professional cloud crawler. The crawler comprises a user management client for defining a crawler task, submitting the crawler task, setting a crawler parameter, checking a crawler running condition and acquiring crawled data, and further comprises a crawler server end for implementing scheduling of the crawler task and processing of the crawled data, and a crawler acquisition client for acquiring internet site data and collecting information of a network bandwidth rate of a running host; the crawler server end comprises distributed scheduling and distributed processing, wherein the distributed scheduling achieves scheduling of the crawler task, management of crawler client resources and reception of data returned from a crawler client; and for the distributed processing, a distributed data processor consumes data in a message queue in real time and processes data in a streaming manner. The crawler provided by the present invention improves the utilization rate of user idle resources, saves the cost for developing a crawler system by a user, and realizes fair sharing of resources.

Description

technical field [0001] The invention relates to the field of network information collection, in particular to a multi-tenant professional cloud crawler based on volunteer computing. Background technique [0002] Now we have entered an era of data explosion. With the development of Internet and mobile Internet technology, the Web has become a platform for data sharing. Then, how to let people find the information they need in the massive data will become more and more difficult. It's getting harder. [0003] Under such circumstances, general search engines (Google, Bing, Baidu, etc.) become the best way for everyone to quickly find target information. When users are relatively clear about their needs, it is very convenient to use a general search engine to quickly find the information they need through keyword searches. However, general search engines cannot fully meet the needs of users for information discovery. That is because in many cases, firstly, general search engin...

AI Technical Summary

Problems solved by technology

[0007] The problem of Internet information collection is a relatively common demand. Internet companies as large as Baidu, Tencent, and Ali, etc., and companies as small as companies that develop weather forecast apps all have the need to crawl data. At present, the solutions for Internet information collection are mainly Each user develops and builds a crawler system according to their own needs, and it is no exception for those SME customers

For small and medium-sized enterprise customers, because most of the crawled sites are relatively few and infrequent, many bandwidth and server resources of the enterprise are idle.

Images