Web page information extraction system and method

Abstract

The invention relates to a system for extracting web page information and a method thereof. The system comprises a template generation module, a web page homogenization module, an automatic tagging module, a wrapper file generation module and an on-line extraction module, wherein, the template generation module is used for selecting web pages to be automatically tagged from a web page collection, and the web pages to be automatically tagged is classified according to training web pages tagged by a user, so as to generate a classified category web page template; the web page homogenization module is used for screening out the difference between the automatic tagging web pages and the web page template belonging to the same category with automatic tagging web; the automatic tagging module is used for analyzing training web pages corresponding to the category, so as to generate a first wrapper file; automatic tagging can be performed on the automatic tagging web pages according to the fisrt wrapper file, so as to generate new training web pages; the wrapper file generation module is used for analyzing all the training web pages and generating a second wrapper file; and the on-line extraction module is applied to the second wrapper document, and is used for extracting unselected web page information in the web page collection. The invention ensures that a plurality of templates corresponding to inhomogeneous web pages can be generated, and extracting can be performed on a plurality of records in a web page and a plurality of attributes of each record.

Description

technical field [0001] The invention belongs to the field of network information processing, and in particular relates to a system and method for extracting web page information. Background technique [0002] The current web page extraction technology can be divided into specific domain-specific web page extraction technology and general web page extraction technology according to the application field. [0003] In the web page extraction technology for a specific field, it is usually necessary to make some assumptions about the content to be extracted. For example, extracting the text of news web pages, extracting some specific attributes in web pages, such as extracting product prices. This type of method often extracts web pages according to the characteristics of the object to be extracted, through statistical methods or through the method of summarizing starting rules. However, due to the particularity of the extracted objects, the generality of this method and the ty...

AI Technical Summary

Problems solved by technology

[0011] First, the current web crawler judges whether these web pages are under the same ur1 path when crawling similar web pages in the website. However, there are a large number of dynamic ur1s on the current website, and even this situation exists. The structure of the ur1 path where the webpage is located may be very different even between the same webpages

This will cause the wrapper file generated by the training webpage to fail to extract the webpages generated by different webpage templates in the webpage collection

[0012] Second, even if these webpages are generated by the same webpage template, but there are many non-template nodes in the webpage, and there are various differences between the non-template nodes of different webpages, then the wrapper files that are only generated by some training webpages often cannot Covering all these differences has caused the wrapper file pair to be incapable of extracting this part of the webpage, and the traditional method is to submit these webpages that cannot be extracted correctly to the user, and let the user mark the data in these webpages area, and then provide these web pages as training web pages to the web page extraction program to regenerate the wrapper

[0013] Thirdly, the current web page extraction systems all have a contradiction between accuracy, automation and the required manual intervention. For example, extraction systems with higher accuracy and fewer training samples often take longer The running time cannot meet the needs of online real-time extraction, but a system with higher efficiency in the extraction stage often requires more training pages and manual intervention to generate wrapper files with better precision and recall

[0014] Fourth, the update speed of the current website is fast. After the correct wrapper file is generated, with the revision of the website, the task of extracting webpages from the revised website cannot be completed by using the wrapper file generated by the old version of the webpage.

[0015] Fifth, at this stage, many web page extraction technologies are aimed at certain types of websites, such as only news web pages can be extracted, or only certain products of a certain object can be extracted, such as only the price and name of the product and other attributes to extract

Images