Method and device for collecting webpage data of direction site based on internet

Abstract

The invention discloses a method and a device for collecting webpage data of a direction site based on internet, in order to solve the problem of the prior art that a file acquisition system cannot be used for timely and efficiently acquiring data of an acquisition site. The method comprises the following steps: according to the priority of to-be-acquired URL (uniform resource locator), adding the to-be-acquired URL into a URL array having the corresponding priority; confirming each URL array weighted value according to the URL amount, the URL array priority value and a weight factor in each URL array, wherein the weight factor is the new URL linkage amount in a list page which is updated for updating and acquiring content page linkage; acquiring the URL from the URL array having the highest weighted value; calculating the URL array weighted value; accessing the URL in the to-be-accessed URL array having the highest weighted value; and acquiring the page data according to the accessed URL, thereby realizing timely and efficiently acquiring data.

Description

technical field [0001] The invention belongs to the technical field of computer Internet, and in particular relates to a method and a device for collecting web page data of an Internet-based directional site. Background technique [0002] The Internet has been in a state of rapid development, and the amount of information has expanded rapidly. More people search for relevant information through the Internet. Although various information can be searched using public search engines, there are many defects in the results of these search engines: real-time performance is not enough, detailed text information cannot be directly seen, etc. As a result, many network acquisition systems have been born. These collection systems generally configure the site to be collected, set the initial URL, and the collection system automatically sets the priority of the URL according to the level of the website structure, and crawls the webpage according to this level. This mechanism basically...

AI Technical Summary

Problems solved by technology

[0004] In order to solve the problem that the prior art file collection system cannot guarantee the timely and effective data collection of the collection site, the embodiment of the present invention provides a method for collecting web page data based on the Internet-based directional site, including:

Images