Method and system for obtaining web page content

Abstract

The invention discloses a method and a system for obtaining webpage content, wherein the method comprises the following steps that: judging if a webpage in a website satisfies a reporting trigger condition; when the webpage satisfies the reporting trigger condition, reporting webpage information through the website; and arranging a crawler to capture the webpage content from the webpage by an allocation server according to the reported webpage information. In the method and the system for obtaining the webpage content, when the webpage satisfies the reporting trigger condition, the webpage information is reported; and the crawler captures the webpage content from the appointed webpage according to the webpage information. Through the method, the work amount of the crawler is reduced; the pressure of the target website is relieved; the ability of obtaining real-time information is improved; and a favorable condition is provided for real-time search.

Description

technical field [0001] The present invention relates to the field of information retrieval, and more specifically, to a method and system for acquiring web page content. Background technique [0002] With the emergence of a large number of blogs, microblogs and other websites on the Internet, users have high requirements for real-time acquisition of network content, and the management of sudden massive information pushes the acquisition of real-time information to the top. most important position. [0003] At present, search engines mostly use crawlers to obtain web page information. As the links contained in the web page are continuously transmitted and obtained, the efficiency of information acquisition is reduced and a lot of resources are wasted. Moreover, because of the huge amount of visits in this way of obtaining web pages, the update of the web page content at the same location cannot be obtained at the first time, and the presentation of real-time information cann...

AI Technical Summary

Problems solved by technology

[0003] At present, search engines mostly use crawlers to obtain web page information. As the links contained in web pages are continuously transmitted and obtained, the efficiency of information acquisition is reduced and a lot of resources are wasted.

Moreover, due to the huge amount of visits in this way of obtaining web pages, the update of the web page content at the same location cannot be obtained at the first time, and it is basically impossible to realize the presentation of real-time information

[0004] Specifically, the following problems that exist in the crawler crawling method have seriously affected the real-time acquisition of web page information: (1) crawler acquisition requires a large number of redundant crawling of irrelevant or repeated webpages, and the efficiency is very low; (2) It is impossible to obtain the content update of the webpage in real time; (3) In order to obtain the information of the same webpage, it is necessary to visit the webpage repeatedly, which puts a huge pressure on the server and bandwidth

[0005] It can be seen that the traditional web page information acquisition method cannot realize the acquisition of real-time information, so it cannot meet the needs of actual use

Images