System and a method for detecting the key content of a web page based on visual characteristics

Abstract

The invention relates to a system and a method for detecting the key content of a web page based on visual characteristics, which collects a web page sample library and utilizes chrome Head less software, dynamic rendering HTML code, and analysis of the visual characteristics of the internal DOM control and the properties of sub-components to form a multi-dimensional feature vector, using decisiontrees, random forests, Bayesian analysis, logistic regression, support vector machines, K-Nearest neighbor algorithm to detect, output the probability of the component as the key content, to achieveautomatic extraction of the key content of the web page. The invention can extract automated key contents for unknown web pages, and is used for search engine to extract abstracts for large-scale webpages.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to an automatic detection system and method for key content of webpages based on visual features. Background technique [0002] With the widespread application of the Internet, web pages have become an important carrier for users to obtain information. When search engines use web crawler software to crawl web pages, they need to analyze the key content, remove non-key content such as advertisements, navigation bars, and user comments in the web page, and provide users with a summary of the target web page. On the other hand, with the complexity and diversification of web design and the further popularization of web page dynamic rendering technology, a lot of key content is often added dynamically through JavaScript code, while the traditional tag analysis based on static HTML code is used to analyze key content The detection method has been unable to adapt to the increasingly ...

AI Technical Summary

Problems solved by technology

However, the shortcomings of this method are that the visual related features are not fully considered in the process of dom feature extraction, the process implementation is too complicated, and the algorithm lacks the ability of self-learning

However, the disadvantage of this method is that the content extraction needs to customize the web page template in advance, once the structure of the web page changes, the success rate of content extraction will be reduced

Most of the proposed solutions are based on the basic features of tags and the semantic features of content to detect key content. However, with the emergence of technologies such as web page overlay and dynamic rendering, it is not easy to identify them only by relying on the basic features of tags and semantic features. key content

Secondly, on the algorithm of machine learning, generally only one algorithm will be selected for testing. Whether it is machine learning or deep learning, it is impossible to improve the accuracy of the overall learning.

Images