Online Web news content extraction system

Abstract

The invention relates to an online Web news content extraction method. The method comprises the following steps: obtaining a DOM tree of an extracted Web news webpage; traversing the DOM tree to construct a text node information sequence and a label path information sequence; calculating a label path characteristic value sequence; fusing the label path characteristic value sequence by a weighted DS evidence theory to obtain a label path comprehensive characteristic value sequence; constructing a text node comprehensive characteristic value sequence; and extracting the text content of the Web news webpage according to the text node comprehensive characteristic value sequence. The invention furthermore discloses an online Web news content extraction system which consists of an analysis module, a calculation module, a fusion module and an extraction module. The label path characteristics are not based on templates of the webpages and have diversity, and the whole extraction process only comprises simple mathematical elements, so that massive heterogeneous Web news webpages can be effectively extracted online.

Description

[0001] The application date is May 10, 2013, the application number is 2013101732801, the title of the invention is: a method and system for extracting online Web news content, and the applicant is a divisional application of Hefei University of Technology. technical field [0002] The invention belongs to the field of network information processing, in particular to an online Web news content extraction method and system. Background technique [0003] With the rapid development of the Internet, Web news pages have become the main platform for people to publish and obtain information after traditional newspapers, radio, and television. At present, in addition to the main content, Web news pages are also mixed with a large amount of information irrelevant to the subject content, such as navigation bars, advertisements, recommended links, copyright statements, and so on. These noise data, which account for 40%-50% of the entire web page data, seriously affect the service quali...

AI Technical Summary

Problems solved by technology

[0011] First, many current webpage extraction technologies assume that the extracted webpage objects are generated by the same webpage template, and the existing wrappers are difficult to effectively extract the content of webpages generated by unknown templates, and their versatility is poor

If you need to extract the content of a web page with an unknown template, you need to build a new wrapper for the template, and any change in the template will cause the wrapper to fail, and the cost of maintaining these templates online is extremely high

Even if the webpages are generated by the same template, there are still many non-template nodes in these webpages, and there are certain differences between the non-template nodes of different webpages. Only the wrappers generated by some training webpages cannot cover these differences. Competent for the extraction task of some web pages

[0012] Second, many current web page extraction technologies are not suitable for online extraction tasks

This type of method is simple in design, single in perspective, and completely ignores the hierarchy of characters in HTML text, and the hierarchy is closely related to the distribution of web content, so it is difficult to be used for the extraction of massive heterogeneous web news web pages

Images