Intelligent extraction system and intelligent extraction method for article type web pages

Abstract

An intelligent extraction system for article type web pages comprises a web page loading module to be extracted, a wrapper query module, a web page extraction module, an unsuccessfully extracted web page collection module, a learning judgment module, a web page learning module and an extraction wrapper management module.

Description

technical field [0001] The invention relates to a system and method for intelligently capturing articles in real time on the Internet, which can be applied to Internet fields such as portal websites and search engine websites that require large-scale, accurate and real-time capturing of articles. Background technique [0002] Internet portal websites have a large demand for reprinting articles every day, and have high requirements for the quality of articles. Many existing crawling systems can meet this requirement, but they all suffer from the following three problems: [0003] 1) The crawling system that uses the machine-automatically generated extraction wrapper technology can capture a large number of articles, but it cannot achieve accurate extraction of articles, and the usability of crawling articles is low; [0004] 2) The article extraction results of the crawling system using the artificially generated extraction wrapper technology are accurate, but it is necessar...

AI Technical Summary

Problems solved by technology

[0003]1) The crawling system that uses machine-automatically generated extraction wrapper technology can capture a large number of articles, but cannot achieve accurate extraction of articles, and the availability of crawled articles is low ;

[0004]2) The article extraction results of the crawling system using artificially generated extraction wrapper technology are accurate, but it is necessary to generate, update and maintain the extraction wrappers for thousands of websites on the Internet Ordinary vertical crawlers cannot take on this job very well, and can only rely on a large amount of human participation;

[0017]1) The extraction system using the machine-automatically generated extraction wrapper technology can capture a large number of articles, but it cannot achieve accurate extraction of articles, and the availability of captured articles is low;

[0018]2) The extraction system adopts artificially generated extraction wrapper technology, and the article extraction results are accurate, but it is necessary to generate, update and maintain the extraction wrappers for thousands of websites on the Internet work, can only rely on a large number of human participation;

Images