Web page element extraction method and web page element extraction system

Abstract

The invention provides a page element extraction method and a page element extraction system. The page element extraction method comprises building a DOM (Document Object Model) tree structure which is corresponding to a webpage; sorting nodes of the DOM tree structure by utilizing a decision tree and building a first block sequence of the webpage according to a sort result; inputting conditional random fields to the first block sequence, performing optimal computation and obtaining a second block sequence; selecting sequence elements of presupposed types in the second block sequence and extracting page elements which are corresponding to the sequence elements. According to the technical scheme of the page element extraction method, the block sequences of the webpage are built according to the DOM tree structure of the webpage, irrelevant contents of blocks are filtered, needed page elements are extracted, manual rules are not needed in the extraction process, and the problems that the manual rules are low in efficient and complex in maintenance are solved.

Description

technical field [0001] The present invention relates to the field of the Internet, in particular to a webpage element extraction method and a webpage element extraction system. Background technique [0002] Generally speaking, web pages contain rich and complex information, including navigation, title, text, time, and even advertisements. In order to extract effective elements from a web page, it is necessary to conduct a detailed analysis of the web page. In the prior art, there are two processing methods for extracting web page elements. [0003] The first is to use manual setting rules to extract elements that are fixed in a certain area of ​​the page. [0004] The second is to manually mark the page compilation language to form a webpage construction template. For most simple webpages, a certain template form can be summed up based on the position of the webpage information. In this way, when extracting a webpage information, only need to follow this The template corr...

AI Technical Summary

Problems solved by technology

[0005] The efficiency of the above two processing methods is low, and there are problems of poor versatility and low universality. Especially at present, the forms of webpages in the Internet vary widely, and the dimensionality of webpage features is getting larger and larger, even reaching hundreds of dimensions. It is very difficult to summarize qualified empirical formulas or create labeling templates in complex web pages

In addition, for webpage features with many dimensions, the manual rules written must be very complicated and the maintenance is very complicated; and when the website is revised, the previous template may become invalid, resulting in deviations in the extracted elements

Images