Method for extracting content of web page

Abstract

The invention provides a method for extracting content of a web page, which comprises the following steps: 1) performing a paragraph merger operation on all characterization nodes in a DOM tree constructed by the web page; 2) optionally selecting a node from a node set bearing the longest paragraph, backtracking from the selected node to a root node of the DOM tree, calculating the difference of punctuation amounts of contexts, which are born between a father node of each node and a subtree thereof and between the node and a subtree thereof, and forming an increment sequence in turn; 3) searching the increment of a first 0 value or a first increment smaller than a former adjacent increment and smaller than or equal to a latter adjacent increment in the increment sequence; 4) if the increment is found, determining that the context born by the corresponding node of the increment and the subtree thereof is the content of the web page; 5), if the increment is not found, determining that the context born by the root node of the increment and the subtree thereof is the content of the web page. The method for extracting the content of the web page has the advantages of high extraction precision, high implement speed, low maintenance cost and excellent adaptability.

Description

technical field [0001] The invention relates to the technical field of network information extraction, in particular to a web page text extraction method. Background technique [0002] The rapid development of the network has brought massive network information, how to extract the required network information has attracted more and more attention. Network information extraction refers to the extraction of structured information from semi-structured documents such as web pages. These pages are often automatically generated by server-side applications. The structured information generated by network information extraction provides the most basic analysis data for important network applications such as network mining and network retrieval. Therefore, the research on network information extraction methods has very important practical significance. [0003] The objects of network information extraction can be divided into two categories: one is to extract specific knowledge in...

AI Technical Summary

Problems solved by technology

Due to the lack of full use of the essential characteristics of web pages, these methods cannot meet the actual needs in terms of extraction accuracy, extraction speed, method adaptability, and flexibility when extracting large-scale data from a large number of different information sources.

Images