Method and device for extracting web page text

Abstract

The invention discloses a method for extracting a text from a web page and a device thereof, and relates to the technical field of networks. The method and the device rapidly and accurately extract the text from the web page. The method comprises the following steps: obtaining a start tag and an end tag of a page segment; determining a start position and an end position of the page segment according to the start tag and the end tag; computing a weight value of the page segment; and extracting the page segment with maximum weight value from the web page to be taken as the text of the web page. The device comprises an acquisition module used for acquiring the start tag and the end tag of the page segment; a segmentation module used for determining the start position and the end position of the page segment according to the start tag and the end tag acquired by the acquisition module; a computation module used for computing the weight value of the page segment determined by the segmentation module; and an extraction module used for extracting the page segment with the maximum weight value computed by the computation module from the web page to be taken as the text of the web page. The technical proposal provided by the invention can be widely applied to network systems and devices for reproducing contents.

Description

technical field [0001] The invention relates to the field of network technology, in particular to a method and device for extracting webpage text. Background technique [0002] With the increasingly fierce competition in the market, if the content of the web pages of major websites only depends on the content of the web pages of this site, the content will appear monotonous, which will not increase the click-through rate, nor can it increase the popularity of the site. In order to increase the click-through rate, it is necessary to diversify the content of the webpage, add more hot topics, etc., and the content of the webpage is reproduced from this. Manual reprinting has slow update speed, low efficiency, and consumes a lot of manpower and financial resources. Therefore, crawler software has become the leading software for reprinting webpage content to quickly and accurately extract the text of webpages. [0003] At present, the methods for crawling software to extract web...

AI Technical Summary

Problems solved by technology

[0005] The canonical extraction method needs to spend a lot of energy to maintain the template extraction rules of major website pages. Because there are too many website page templates on the Internet, and the page templates are updated frequently, users cannot maintain these template rules in a timely and accurate manner, and cannot timely maintain these template rules. , accurately extract the text of these web pages

Images