A method for identifying link blocks of web pages based on block tree

Abstract

The invention proposes a block tree-based identification method of Web page link blocks, and based on this, proposes link block discrimination and evaluation indicators, and combines the block tree to propose two basic areas: forward link block discrimination and reverse link block discrimination Block traversal and discriminant algorithm; forward discriminant method has fast recognition speed and large link block granularity, which can be used in the analysis of index type Web pages and text extraction applications; reverse discriminant method can easily and flexibly control the granularity and quantity of link blocks , to ensure the refinement and integrity of the link block, so as to finally realize the comprehensive coverage of the link block on the page link. This method can be used not only in occasions where the link block requires fine granularity, but also in page denoising, text extraction, and page extraction. Automatic generation of templates and other occasions; the block tree proposed by the present invention is used as the basis of Web page analysis and processing, combined with the two proposed traversal and discrimination methods, it can be widely used in the fields of Web data preprocessing and data mining.

Description

technical field [0001] The present invention relates to the fields of web page importance calculation, web page denoising, subject-related link block extraction, web text identification, web page text extraction, refinement of search engine processing unit granularity, and massive web data preprocessing. A method for identifying link blocks of a Web page in a block tree. Background technique [0002] The World Wide Web is a huge network built on links, and links are the soul of the World Wide Web. Every web page in the World Wide Web finally constitutes the most complex network in the world through the links between web pages. Web crawlers also rely on the links between web pages to finally crawl network data. The importance of web pages Sex is also often obtained through link analysis. The number of links in a web page is often between tens and thousands, especially in index (catalogue) type web pages, links account for nearly 100% of the ratio. Although there are many l...

AI Technical Summary

Problems solved by technology

[0011] The second is to ignore some characteristics of non-linked text, such as dates, numbers, unlinked information source annotation text, some special symbols, etc.

In many link blocks, there are a large number of other non-link content such as dates before or after the link, which greatly interferes with the accurate identification of link blocks.

[0012] The third is that the discriminant method is extremely sensitive to the length of the link text when the length of the entire text is not long enough, that is, a certain threshold works well on some pages using short link text, but it is extremely difficult when encountering pages with long link text. possible misjudgment

It is an extremely common phenomenon that the link text lengths of different websites or different webpages vary greatly, which brings great uncertainty to the accurate judgment of link blocks. If the link text becomes shorter, it is likely to mistake the link block. unlinked block

[0013] The fourth is that the error caused by the block problem will affect the accurate identification of link blocks. It is especially easy to identify text blocks that are not separated by block-level elements between the body of the text and link blocks but are in the same block-level element node as link blocks, or Misidentifying link blocks as non-noisy links in the body text

[0014] The fifth is to ignore the number of links

Then obtain the vector containing these four feature values ​​through training, and then realize the block type judgment, but in the face of complex networks, the four features designed here are not universal.

Images