Dynamic network crawler based on client end /service end

Abstract

The invention fulfills reconstruction and design on a network creeper based on local servers on the background of traditional search engines, so as to perform partition on the network creeper into a network creeper client end and a network creeper service end. The service end is installed in the local server on the search engine and has the same function with creeping-decomposed web pages in the traditional network creeper; however, the invention increases a virus identification function, so that only safe web pages can be saved into local databases. The client end is bound to install on a key node in an agency server that a web content supplier or a user uses to accesses Internet, so as to find updates on web page URLs and contents in time, and meanwhile, automatically send the updated URLs to the service end through an information system. The invention has the following beneficial technical effects: The invention can more effectively improve the quality of updating and safety of web pages in local databases in search engines, so as to avoid dead link, link loss as well as being infected by virus web pages with virus while a user is searching the web pages with the search engine, so as to resolve the shortcomings in prior network creepers.

Description

technical field [0001] The invention relates to the field of web crawlers (also known as web spiders, web crawlers or web robots) for computer search engines, in particular to the technical proposal of a web crawler capable of intelligently discriminating and selecting texts and notifying a local database in time to update webpages. Background technique [0002] A web crawler is a background program of a search engine used to discover, explore, and detect web content on the World Wide Web (WWW). The World Wide Web (WWW) is an associative collection of Hypertext Markup Language (HTML) pages distributed on many hosts in the Internet. The pages are linked and accessed through Uniform Resource Locators (URLs), which are the addresses of HTML pages. . At present, the technology of the web crawler of traditional search engine is: web crawling (downloading of webpage) is done by many centralized or distributed crawlers. A URL server sends the list of URLs to the crawler. The cap...

AI Technical Summary

Problems solved by technology

[0003] In order to overcome the shortcomings of existing web crawlers that cannot automatically identify webpages, automatically update webpages dynamically, and cannot give early warnings to webpages containing virus codes, the present invention provides a new construction method for web crawlers. Interact with the server to communicate with the server, automatically notify the server of the search engine when the webpage is updated, and use the embedded anti-virus program to delete illegal webpages