Webpage core block determining method based on DOM (Document Object Model) node text density

Abstract

The invention relates to a webpage core block determining method based on a DOM (Document Object Model) node text density, which comprises the following steps of: 1, analyzing an HTML (Hyperlink Text Markup Language) webpage, and generating a DOM tree to ensure that each HTML label corresponds to one node in the DOM tree, wherein character contents in the webpage are leaf nodes of the DOM tree; 2, adding statistical information including number of all text characters included by the nodes, number of all labels included by the nodes, number of all hyperlink text characters included by the nodes and number of all hyperlinks included by the nodes, and defining text density of the nodes according to the statistical information; and 3, determining a webpage core block according to the text density of the nodes in the DOM tree. According to the invention, on the premise of being free of being influenced by webpage coding styles and remaining the original webpage DOM structure, the core content block in the webpage is completely extracted.

Description

technical field [0001] The invention relates to a web page core block determination algorithm based on DOM node text density, and belongs to the technical field of computer applications. Background technique [0002] With the rapid development of the Internet, WWW has become the largest database in the world. Therefore, data mining in the web to obtain useful information or knowledge has gradually become a new hot research direction. [0003] These studies need to collect, process and store core content in the web quickly and efficiently. However, these core contents in web pages are often surrounded by a large amount of irrelevant information. For example, navigation menus, sidebar ads, copyright information, etc. Although this information can make the content of the webpage rich and beautiful, and at the same time facilitate users to browse, it is not related to the theme of the webpage, which also makes it difficult for these webpages to be parsed by computer programs....

AI Technical Summary

Problems solved by technology

However, this effect cannot be completely eliminated

[0011] 2. Only the text information of the core block of the webpage can be extracted, and the original structural information of the webpage cannot be retained

This makes it difficult to integrate with other applications, such as structured information extraction

[0012] 3. The characteristics of noise data in web pages are not fully utilized, and the distinguishing effect is not very obvious

[0013] 4. The content of the core block cannot be completely extracted, and the rows with low density are easily lost

Images