Subject area identifying method based on weight of text structure

Abstract

The invention provides a subject area identifying method based on weight of a text structure. Based on a tree structure, the method provided by the invention comprises the following steps: resolving a webpage into a label tree and improving a data area excavation and semanteme link block identifying technology on the basis of tree matching, thereby realizing pre-processing for removing links; providing a concept of the weight of the text structure and identifying a subject area with the assistance of the calculation result of the weight of the text structure; and finally adopting a normalization method to calculate a relative length value of a text node in the subject area, and using the normalized relative length value for effectively removing the text node independent from a subject content, thereby realizing denoising in the subject area and acquiring an accurate subject content. By applying the technical scheme of the invention, valuable information on the webpage can be accurately and rapidly excavated, thus the subject area identifying method provided by the invention has a wide application prospect.

Description

technical field [0001] The invention relates to a Web information extraction technology in the field of Web data mining, in particular to a method for extracting text content consistent with the subject of the Web page for text-based semi-structured Web pages. Background technique [0002] Currently, Web information extraction can be divided into the following three ways according to the degree of automation: [0003] (1) Manual method: This method is to manually observe the characteristics of the Web page, then manually mark it, extract the pattern of the target information, and then write a program to generate a wrapper (Wrapper) according to this pattern, and then pass the Wrapper Extract target information. This method can only be used for specific sites and is not universal. Such systems require users to have a solid foundation in computer programming. Because of this, the manual method is suitable for a small number of sites, but cannot be used for a large number of...

AI Technical Summary

Problems solved by technology

Moreover, most of the existing fully automatic extraction technologies use artificial intelligence and machine learning methods. These methods have a large amount of calculation and slow extraction speed, which affects the actual application effect.

In addition, this kind of method often needs to add some prerequisites. For example, RoadRunner needs to provide two pages generated by the same template, and requires the pages to contain repeated patterns, which limits its use.

Images