Method for extracting regular noise from single record web pages

Abstract

The invention provides a method for extracting regular noise from single record web pages. The method comprises the steps of firstly converting multiple record web pages into document object model (DOM) trees, and classifying the DOM trees according to structures; then aligning and integrating the DOM trees belonging to the same type to obtain site section style trees; and positioning approximate positions of web page text headline nodes and approximate positions of web page text main body nodes in the site section style trees, and finally extracting the regular noise in front of texts, in the texts and after the texts according to the web page text headline nodes and the web page text main body nodes. By means of the method, space resources required by construction of the site section style trees is decreased, possible extraction leakage situations are decreased, and extracting speed is accelerated. In addition, an extracting result has high accuracy, good effect is obtained, and the reliability is high.

Description

technical field [0001] The present invention relates to the field of network information retrieval, and more particularly, relates to a method for extracting the front and rear of the text from a single-record web page (that is, a web page of a single style is recorded with a piece of data, and the data record refers to the area of ​​the main part of the web page). Methods for regular noise in and after the text. Background technique [0002] In the information age, there are more and more ways to obtain information. As a carrier of information, the Internet has an irreplaceable position in terms of dissemination efficiency and information capacity. At present, the Internet has become an important source for people to obtain various knowledge and information. However, with the rapid development of Web technology, the massive data information on the Internet is increasing in stages every day, and the content of the information is all-encompassing and in various forms. Web ...

AI Technical Summary

Problems solved by technology

[0012] (1) With the development of web technology, even for the same website, the differences in DOM tree structure of different sections are getting bigger and bigger. The SST tree will cause too many branch nodes in the tree building process, and the number of DOM trees with different structures is not distributed. In the case of uniformity, the occurrence frequency of a certain noise branch in the SST tree is too small, resulting in the phenomenon that only part of the noise can be extracted from web pages with this type of DOM structure;

[0013] (2) Even if the web pages are classified according to the DOM structure, and the algorithm is applied to a DOM tree with a similar structure, if only one node is different in a certain layer of nodes (for example, there are 10 nodes), the SST method will be different for different branches. Establishing different child nodes will cause a lot of waste of space, and the efficiency of tree building will also be greatly reduced;

[0014] (3) When the SST method forms sub-nodes of different styles for different branches, it is easy to cause the branch granularity to be too large, resulting in some small noises and missing extractions;

[0015] (4) Especially for single-record pages, the SST method cannot locate the relative position of the noise and the subject part

[0017] To sum up, for the regular noise extraction in a single record page, the existing noise extraction methods have the problems of missing extraction, waste of space and low efficiency, and cannot locate the relative position of the regular noise and the topic part

Images