A Method of Automatically Extracting List Pages

Abstract

The invention relates to a method for automatically extracting a list page, comprising the following steps: parsing the source code of a web page into a DOM tree; extracting the element node path with text in the DOM tree; scoring and filtering the position information of the element node; extracting the similarity of the node Fingerprint; extract the deep fingerprint of the node block; extract the similar fingerprint of the title and address link; extract the list page and return the encapsulation result. The present invention is applicable to the extraction of a large number of Internet website list pages, and has universal applicability to a large number of list pages. Therefore, even if there is a website revision, the extraction method based on the webpage structure can still take effect, saving the time spent on rewriting the extraction rules and maintenance rules. The resulting time cost and labor cost. In the webpage-based structure extraction algorithm, the location pixel information of elements on the webpage is added as a feature, which is more in line with people's sensory judgment of list pages, making the extraction results more in line with the target.

Description

technical field [0001] The invention relates to the field of network technology, in particular to a method for automatically extracting list pages. Background technique [0002] The traditional list page extraction technology is mainly in the form of rules, such as regular expressions, xpath, css selectors, or even manually collecting the information on the page. [0003] A single web page can accurately collect the desired information through methods such as regular expressions, and the essence of methods like regular expressions and css selectors is to summarize the rules of the source code of the web page by human beings, and then use these rules to Extraction, this method is difficult to extract with the same set of rules on web pages with different structures, because different web pages need different rules to support the extraction. When users need to collect a large number of web pages, they need to rely on manual writing. Rules, this kind of efficiency is not only ...

AI Technical Summary

Problems solved by technology

[0003] A single web page can accurately collect the desired information through methods such as regular expressions, and the essence of methods like regular expressions and css selectors is to summarize the rules of the source code of the web page by human beings, and then use these rules to Extraction, this method is difficult to extract with the same set of rules on web pages with different structures, because different web pages need different rules to support the extraction. When users need to collect a large number of web pages, they need to rely on manual writing. Rules, this kind of efficiency is not only low, even on thousands or tens of thousands of websites, it has become completely impossible to rely solely on manual labor.

Not only that, the extraction method based on rules is limited by the webpage itself. When the website is revised, the original rules will no longer be applicable, and the rules need to be rewritten manually. This also makes some projects that rely on open source information collection Maintenance costs become insanely high

Images