Method for extracting webpage text based on label path and text punctuation ratio feature fusion

Abstract

The invention discloses a method for extracting a webpage text based on a label path and a text punctuation ratio feature fusion. The method is mainly by constructing a text punctuation ratio and a feature fusion method of a label path to propose a novel feature value, thereby extracting the text from a webpage. The method is characterized in that a text punctuation ratio feature pair is defined to measure the average sentence length of the label path, and at the same time, the position of the label path and its internal complexity are combined to give a more comprehensive feature value to judge the content of the text. By adoption of the method, it is possible to extract the webpage text more accurately without constructing an extraction template, and the application scope is wide.

Description

technical field [0001] The invention relates to the technical field of data mining and web page analysis, in particular to a web page text extraction method based on the fusion of label path and text punctuation ratio features. Background technique [0002] With the rapid development of the Internet, the Internet has become an important channel for people to obtain information, and people are also accustomed to publishing various information on the Internet. What follows is that the amount of web pages and information in the Internet grows at a very high speed. In addition to the text information, the web page also contains advertisements, navigation bars and other content that has nothing to do with the text. In order to enable users to quickly obtain useful information, the text extraction method also came into being. [0003] A web page contains a lot of content, but only a limited part of the content is really needed by users, and this part of the content is called the...

AI Technical Summary

Problems solved by technology

There are some problems in the typical text extraction method: 1. Different templates need to be established for different web pages; 2. When the structure of the web page changes, the template is no longer applicable; 3. Using machine learning methods to construct templates requires a large number of training web pages ; 4. The computational complexity may not be suitable for online systems; 5. The number of features and content of the current feature-based extraction methods are relatively complex, or have strong pertinence; 6. The current extraction methods cannot be applied to web pages containing a single The case of body blocks and multiple body blocks; that is, in the current method, it is not suitable for most web page structures

Images