Webpage information extraction method and device

Abstract

The embodiment of the invention provides a webpage information extraction method and device. The webpage information extraction method comprises the steps of determining candidate elements and text information of a webpage according to a DOM tree of the webpage; determining candidate position information of the webpage according to the display page screenshot of the webpage; determining a first probability that each candidate element serves as a target extraction element and a second probability that each candidate position serves as a target extraction position according to the candidate position information and the text information of each candidate element; determining a target extraction element and a target extraction position from each candidate element and each candidate position according to the first probability and the second probability; and performing information extraction on the webpage according to the candidate elements determined to be the target extraction elements and the candidate positions determined to be the target extraction positions. The text information and the candidate position information of the candidate elements of the webpage are extracted, and theneural network model and the spatial probability distribution model are used for positioning the extraction position, so that the positioning precision and the fault-tolerant rate of webpage information extraction are improved, and automatic extraction of webpage information is realized.

Description

technical field [0001] Embodiments of the present invention relate to the field of Internet technologies, and in particular, to a method and device for extracting web page information. Background technique [0002] With the rapid growth of information in the Internet, Internet pages have become the most important way for humans to acquire knowledge and information. Traditional search engine technology can quickly sort web pages according to user queries, improving the efficiency of information retrieval. However, for a large number of results fed back by search engines, manual investigation and screening are also required. With the explosive growth of information, this information retrieval method has been difficult to meet the needs of people to fully control information resources. The emergence of knowledge graph technology provides new ideas for solving information retrieval problems. The knowledge graph technology returns the processed and recommended knowledge to user...

AI Technical Summary

Problems solved by technology

[0003] Embodiments of the present invention provide a method and device for extracting webpage information, which are used to solve the problem of low efficiency and accuracy of the method of relying on manually marking the DOM tree to locate the location of webpage information extraction

Images