A system capable of identifying and automatically collecting web page information

Abstract

The invention discloses a system and a method for identifying and automatically acquiring webpage information. The system comprises an anti-acquisition classifier constructing module, an automatic acquisition identifying module and an anti-acquisition online processing module, wherein the anti-acquisition classifier constructing module is mainly used for automatically acquiring history web information by using a computer program and learning and distinguishing normal webpage access behaviors; the automatic acquisition identifying module is used for automatically identifying the automatic acquisition behavior of a search engine program by using an anti-acquisition classifier in the previous step, and adding an IP (Internet Protocol) segment where the identified acquisition program is positioned into a blacklist; and the anti-acquisition online processing module is mainly used for automatically judging and processing accessing users on line. Due to the adoption of the system and the method, the deficiencies in the prior art are overcome; and in the system, the history webpage access behaviors of a website are analyzed, the automatic acquisition classifier is established, automatic acquisition of a robot is identified, and webpage anti-grabbing is realized through automatic robot acquisition and identification.

Description

technical field [0001] The invention relates to the technical field of dynamic analysis of webpages, in particular to a system capable of identifying and automatically collecting webpage information. Background technique [0002] With the development of the Internet, there have been more and more Internet sites in endless forms, such as news, blogs, forums, SNS, Weibo and so on. According to the latest statistics from CNNIC this year, China now has 485 million Internet users and more than 1.3 million domain names of various sites. In today's Internet information explosion, search engines have become the most important tool for people to find Internet information. [0003] Search engines mainly crawl website information automatically, preprocess it, and build indexes after word segmentation. After entering the search term, the search engine can automatically find the most relevant results for the user. After more than ten years of development, the search engine technology ...

AI Technical Summary

Problems solved by technology

[0007] 2. The information on some websites has privacy or copyright, and many webpages contain information such as background databases, user privacy, passwords, etc.

Parsing process such as figure 2 As shown, the method of building dynamic webpages with dynamic technology only increases the difficulty of webpage collection and analysis, and does not fundamentally eliminate the collection of search engines.