Webpage purification system based on Render_DOM model and purification method thereof

Abstract

The invention provides a webpage purification system based on a Render_DOM model and a purification method thereof. The webpage purification system comprises an acquisition module, an analysis module, a rendering module, an optimization module and a view display module, the analysis module is used for generating a Render_DOM rendering tree, and each tree node renderer of the RenderDOM rendering tree is embodied as a corresponding DOM visual node and a rectangular frame of a CSS style rule calculated for the DOM visual node; a block element of a tree node renderer is set as a Block node; the Block block tree module is used for forming a Block block tree by taking Block nodes of the Render_DOM rendering tree as numbering paths in a sequence from top to bottom and from left to right, wherein the Block block tree comprises basic unit blocks; the merging module is used for merging the nearest basic unit blocks on the Block tree until the area of the rectangular frame corresponding to the basic unit blocks reaches a page block threshold value; and the segmentation module is used for deleting the basic unit blocks of which the similarity exceeds a similarity threshold value as repeated contents. The working efficiency is improved.

Description

technical field [0001] The present application relates to the technical field of webpage purification, in particular to a system and method for purifying webpages based on the Render_DOM model. Background technique [0002] Nowadays, when people obtain information on the content of web pages on the Internet, in addition to encountering some high-quality web pages that only contain content related to the topic, they often encounter web pages with a lot of network noise pollution. Noise pollution, that is, Wed pages contain a lot of irrelevant content. The existence of this kind of webpage noise not only occupies the normal webpage space, makes the webpage confusing, blocks the normal document flow layout, but also visually interferes with people's acquisition of topic information. Therefore, we need to purify webpages polluted by network noise. In the prior art, webpage segmentation technology is generally used to purify webpages, and the noise-polluted parts are segmented. ...

AI Technical Summary

Problems solved by technology

And these methods are either based on the DOM tree model alone, or based on the cssom tree model alone, and only consider a single model algorithm

Specifically, the algorithm using the dom tree model is classified according to the function of the semantic tags of html, and then the corresponding dom tree is extracted. All nodes have relationships with each other. It can build a tree structure according to the semantics of html documents. Corresponding the visible and invisible elements that exist on the page, interpreting the entire page structure and content, but only focusing on semantic tags, resulting in excessive construction of unnecessary dom elements, such as some that have little to do with the theme The tags , , etc., lead to extremely redundant DOM tree construction, and do not consider the layout changes caused by style elements; while the algorithm using the cssom tree model uses a nested box model to locate the web page The position and display mode of the elements in the webpage, each element in the webpage is positioned and laid out through the box model, and the cssom tree model algorithm only considers the visual representation of the webpage, while ignoring the semantic content of the page

Therefore, none of these methods can achieve the unification of web page semantics and visualization, and it also shows that using a single model cannot fully and reasonably divide web pages.

Images