Web page denoising method and system based on cooperative work of template and classifier

Abstract

The present invention discloses a web page denoising method and system based on templates and classifiers working together, wherein the denoising method includes: parsing the acquired original HTML document, deleting irrelevant label nodes, and generating a simplified DOM that meets requirements tree; calculate the characteristics of each block-level node in the DOM tree of the target web page, and obtain the original node set; add the original node set to the cache node set of the corresponding website, and trigger the template when the number of elements in the cache node set reaches the preset threshold Generate an algorithm to update the template node set of the corresponding website; use the template node set of the website to which the target web page belongs to filter the original node set of the target web page to obtain the filtered target web page node set; use the trained classifier to classify the filtered target web page nodes The collection is classified, and the node whose classification result is the main content is reserved, from which the main content text is extracted. The invention has less manual intervention and high efficiency, and is suitable for denoising various theme webpages.

Description

technical field [0001] The invention relates to the technical field of webpage denoising, in particular to a webpage denoising method and system based on templates and classifiers working together. Background technique [0002] With the continuous development of Internet technology, the amount of information on the Internet is getting bigger and bigger, showing explosive growth. Massive web page information is the main embodiment of Internet information, and it is a natural data mine for many other research fields, including: search engines, public opinion analysis, natural language processing, etc. However, in addition to the main content of web pages, there are also some commercial advertisements, navigation bars, copyright information, announcements and other information that are not related to the main content. These information can be called web page noise. How to remove noise content in web pages and extract web pages The main content is for the analysis and use of th...

AI Technical Summary

Problems solved by technology

The rule-based method is to formulate some heuristic rules in advance to filter out those text contents that meet the rules. This method is only applicable to some simple web pages, and complex heuristic rules are required for web pages with complex structures, which has its limitations.

The template-based method has a fast denoising speed, but it often needs to manually construct a template suitable for a specific website page, which cannot be used as a general webpage denoiser. In 2010, Li Liwen et al. In "Research", webpage similarity calculation is used to classify different webpages, and a corresponding template is constructed for each class. The template uses the location information of the main content. ) nodes, select the nearest parent node containing these main body content as the template, the proposed main body information may contain a lot of noise, which has a great impact on the denoising effect

The denoising method based on visual content first divides the webpage into different blocks, uses artificial annotation and predicts the importance of the webpage block through neural network and support vector machine, and finally selects the most important webpage block, but the method calculates Large volume, low efficiency

Images