Webpage information acquisition system and method

Abstract

The invention provides a webpage information acquisition system and method. The system comprises a task delivery device, a task queue, a task actuator and a memory. All the modules are independently operated in a service way, and this property guarantees good expandability of the whole system. The method comprises the steps of packaging a URL into a task to be downloaded, delivering the task to the task queue, receiving the task through the task queue, acquiring the task from the task queue, acquiring a corresponding task resolver according to the task type, analyzing downloaded webpage source code through the task resolver, storing data, packaging a sub-URL analyzed in a URL webpage into a new task, and delivering the new task to the task queue again. The system and the method can applied to webpage information download in any vertical fields, can easily add different website resolvers by paying attention to custom only, so as to guarantee the accuracy of webpage information acquisition of the whole network, and systems with universal repeated download and update frequencies are easily controlled.

Description

technical field [0001] The invention relates to the field of network video information acquisition, in particular to a web page information acquisition system and method. Background technique [0002] At present, web page information acquisition technologies mainly focus on the acquisition of web page information on the entire network and the acquisition of vertical web page information. [0003] Ordinary web search engines, such as Google's Google Search (www.google.com) and Baidu's Baidu Search (www.baidu.com), this type of web search, its information is mainly from each download page according to the Certain conditions extract important information such as text, but this extraction can allow less accurate information (for example: in a video playback page, it will not pay too much attention to who the director is and how long the video is. Another example: TV shows In the preview page, it does not need to pay attention to whether a program starts at 7:00 or 7:30), and ve...

AI Technical Summary

Problems solved by technology

Therefore, the accuracy of information obtained from the general network-wide web page information cannot meet the requirements

[0004] The current webpage acquisition system mainly solves how to obtain the webpages that the system needs to obtain, but the accuracy of the information in its own pages cannot meet the requirements of the video vertical field; the video vertical field requires information update frequency. Fetching pages cannot precisely control how often they are updated

[0005] Currently, there is no general-purpose system and method for obtaining vertical web page information with a clear structure, independent functions, and easy control of repeated downloads and update frequencies.

Images