Web page content extracting method and device

Abstract

The invention provides a web page content extracting method and device.The method comprises the steps that characteristic values of all text sections in web page HTML text subjected to denoising preprocessing are calculated, multi-time aggregation is conducted on text, meeting the requirement, in a text section list according to different aggregation rules, and web page content meeting the requirement is obtained finally.The method is simple and efficient and avoids the cumbersome process for artificially making extracting rules for the full text; automatic content extraction of a relevant web page can be achieved according to a specific web page type.

Description

Technical field [0001] The invention relates to the technical field of word crawling, in particular to a method and device for extracting the body of a webpage. Background technique [0002] With the rapid development of Internet technology, web pages have become an important source of information for people. However, unlike traditional text, in addition to valid information (text), web pages also contain a lot of invalid information (noise), such as website navigation links, advertising content and links, copyright information, etc. This greatly reduces the efficiency of obtaining valid information. In order to effectively use the information on web pages, a fast and accurate automatic text extraction method is needed to filter invalid information and extract the text content that users really need. This also affects public opinion monitoring and analysis, and analysis results based on Internet big data mining. Key factors. [0003] The existing web page body extraction methods ...

AI Technical Summary

Problems solved by technology

The method based on the DOM tree parses the HTML file of the web page into a DOM tree, and determines the position of the text according to the distribution law of the text information in the DOM tree. Big differences, different complexity, specific distribution rules can only be effective for some webpages, generalization ability is average

The method based on template rules generally manually configures the extraction template rules for a specific website, so that the text can be quickly extracted in the specified area, but the irregular update of the website often leads to the failure of the template and the inability to effectively extract information. The extracted text often contains different degrees of impurities (mainly non-standard scripts). This method requires a lot of manual input and high-quality template rule selection to maintain a high extraction accuracy. times are great challenges

[0004] The existing method for extracting webpage text based on the DOM tree structure has the following disadvantages: 1) the construction and traversal efficiency of the DOM tree is low and the speed is slow; 2) for irregular web pages, there may be a risk of failure to construct the DOM tree; 3) the The text distribution of the method is not general, and it only has a good extraction accuracy for some web pages

[0005] The existing methods of extracting text based on template rules have the following disadvantages: 1) The manual maintenance of template rules is heavy; 2) The timeliness of updating template rules cannot be guaranteed; 3) The quality of template rules is difficult to guarantee

Images