Web page content extraction method and web page content extraction system

Abstract

The invention provides a webpage content extraction method and a webpage content extraction system. The web page content extraction method includes: extracting the DOM tree structure corresponding to the web page; traversing the DOM tree to obtain the dimensional features of each node in the DOM tree; inputting the dimensional features of each node into a decision tree according to predetermined rules, and classifying each node , and determine the structural blocks of the webpage according to the classification results of the decision tree; and selectively extract the corresponding webpage content according to the structural blocks. Utilize the technical scheme of the present invention, carry out structural block according to the DOM tree structure of webpage, filter out the content of irrelevant block according to structural block, extract the required block webpage content, do not need to use manual rules in the process of block and extraction , which solves the problems of low efficiency and complicated maintenance of manual rules.

Description

technical field [0001] The present invention relates to the Internet field, in particular to a web page content extraction method and a web page content extraction system. Background technique [0002] Generally speaking, web pages contain rich and complex information, including navigation, title, text, time, and even advertisements. In order to extract effective content from a web page, fine analysis of the web page is required. In the prior art, there are two processing methods for extracting web page content. [0003] The first is to use manual setting rules to extract the content of a fixed area in the page, [0004] The second is to manually mark the page compilation language to form a webpage construction template. For most simple webpages, a certain template form can be summed up based on the position of the webpage information. In this way, when extracting a webpage information, only need to follow this The template corresponding to the web page extracts the corre...

AI Technical Summary

Problems solved by technology

[0005] The efficiency of the above two processing methods is low, and there are problems of poor versatility and low universality. Especially at present, the forms of webpages in the Internet vary widely, and the dimensionality of webpage features is getting larger and larger, even reaching hundreds of dimensions. It is very difficult to summarize qualified empirical formulas or create labeling templates in complex web pages

In addition, for webpage features with many dimensions, the manual rules written must be very complicated and the maintenance is very complicated; and when the website is revised, the previous template may become invalid, resulting in deviations in the extracted content

Images