Method for acquiring news web page text information

Abstract

The invention relates to a method for extracting the text information in a new webpage, belonging to the field of webpage information analyzing and processing technique. The existing technique ordinarily adopts a packer to extract the interesting data from the web pages and the obtaining of information mode recognition knowledge of the packer is a time-taking and laborious, higher intelligence-demanding operation. The invention uses a stack data structure to convert the hiberarchy information of webpage data into vectors, constructs and analyzes Html tree, then making compression on the data of various layers of the Html tree, making data filtration, thinning, recognition and recombination to extract the needed data information. The invention is applied to extracting the template-generatednews information in the news web pages from a fixed website.

Description

technical field [0001] The invention belongs to the technical field of webpage information analysis and processing, and in particular relates to a method for extracting news webpage text information. Background technique [0002] With the rapid development of the Internet, the amount of information on the Internet, that is, the Web, is increasing at an alarming rate every day. Many companies often need various information and usually collect information on a large scale from the Internet. Therefore, the collection of massive information has become a concern for every enterprise. The problem. Because the current information processing technology is aimed at content in plain text format, and the information on the Web mainly exists in the form of static Html, how to convert the information in the form of Html collected on the Web into valuable text format information , to facilitate subsequent information processing has become an urgent technical problem to be solved. [000...

AI Technical Summary

Problems solved by technology

[0010] (1) The text information extracted by this method is incomplete: because the text information of news web pages not only exists in the table, but also exists in the div; in addition, the information of the news includes not only the text information, but also the title information. Extraction of title information, this method does not involve

[0011] (2) The text information extracted by this method is not accurate enough, and the efficiency is not high: because the threshold setting in the method of selecting candidate table nodes is difficult to grasp, the size of the threshold has a great influence on the extraction of text information, so if the threshold is set If it is not appropriate, the extracted text information will be very inaccurate; even if a suitable threshold is selected, only the table nodes whose number of Chinese characters contained in the string is greater than the threshold are used as candidates. This method of extracting text information is Not accurate enough

In addition, the setting of the threshold requires a large number of experiments, which also affects the extraction efficiency.

Images