Web text information extraction method

Abstract

The invention provides a web information extraction method aiming to solve the problems of high complexity and low accurate rate of an existing information extraction method. The method includes the steps that an HTML source code is obtained and loaded to a first memory zone; an HTML parser is built, the HTML source code is parsed in combination with page label distribution, and parsed data are stored to a second memory zone; unit windows are defined, the unit windows are moved to conduct primary feature statistics on data in the windows, voice variance and text similarity are sequentially adopted to serve as secondary features to further remove noise, and the density quantitative value of all the windows is obtained; the relation between the density threshold and the text density of all the windows is concluded according to sample data, and a reasonable extraction scheme is formulated; finally, a text integration module is entered, and a text with a normative format is output. Above all, a loading-parsing-quantizing-selecting-outputting processing scheme is adopted for the HTML source code, different web pages are adapted automatically according to the text density relative value, a great number of web pages can be processed, and webpage information is extracted automatically.

Description

technical field [0001] The invention relates to an information processing technology, in particular to a text information extraction technology of HTML source code, and belongs to the field of Internet information processing. Background technique [0002] With the popularization of Internet technology, the data on the web has increased rapidly, and the information carried on the web page has become an important source of human information. However, a large number of so-called "noises" such as web links, advertisements, and plug-ins are embedded in web pages, which makes the complexity of information processing such as information retrieval, data mining, machine translation, and text summarization soar. In this context, there is an urgent need for a fast and effective web text extraction method to remove the noise outside the text of the web page and correctly extract the text information of the web page. [0003] Among the currently commonly used information extraction algo...

AI Technical Summary

Problems solved by technology

[0004] Aiming at the deficiencies of the existing information extraction algorithms, the present invention provides a multi-feature webpage text extraction method based on text density distribution. This method aims at the text distribution density of the web interface, and first analyzes and integrates the text in combination with the page label distribution, and then uses The unit window is the basic unit to conduct a preliminary feature analysis of the data in the window, and then use the noise variance and text similarity as the secondary features to further eliminate noise and purify the text, and then achieve accurate text extraction

Images