System and method for real-time intelligent capturing of article

Abstract

The invention discloses a system for real-time intelligent capturing of an article. The system comprises a real-time capturing module, a webpage extraction system, a similar document duplicate-removing module, a document automatic classification module and an article publishing module. The real-time capturing module further comprises seven modules running online: a task extraction module, a task analysis module, a task capturing time range test module, a task capturing time interval test module, a task scheduling module, a task downloading module and a task capturing frequency regulation module; and the real-time capturing module still comprises three modules running offline: a task capturing time range discovery module, a task capturing time internal discovery module and a nonprofit agent collection and authentication module.

Description

technical field [0001] The invention relates to the fields of crawling technology, web mining technology, information extraction technology, and natural language processing technology in Internet technology; it can be applied to Internet fields such as portal websites and search engine websites that require large-scale, accurate, and real-time crawling of articles. Background technique [0002] Internet portal websites have a large demand for reprinting articles every day, and have high requirements for the quality of articles. Many existing crawling systems can meet this requirement, but they all suffer from the following three problems: [0003] 1) The crawling system that uses the machine-automatically generated extraction wrapper technology can capture a large number of articles, but it cannot achieve accurate extraction of articles, and the usability of crawling articles is low; [0004] 2) The article extraction results of the crawling system using the artificially ge...

AI Technical Summary

Problems solved by technology

[0003]1) The crawling system that uses machine-automatically generated extraction wrapper technology can capture a large number of articles, but cannot achieve accurate extraction of articles, and the availability of crawled articles is low ;

[0004]2) The article extraction results of the crawling system using artificially generated extraction wrapper technology are accurate, but it is necessary to generate, update and maintain the extraction wrappers for thousands of websites on the Internet Ordinary vertical crawlers cannot take on this job very well, and can only rely on a large amount of human participation;

When high real-time crawling is required, it is necessary to frequently initiate link and download requests to the crawling website server, which will put a lot of pressure on the server of the other party, which in turn will cause the other party to adopt blocking strategies such as denying access to ensure the server works fine, this will cause the fetch to fail

At the same time, high real-time capture requirements consume a lot of network, server and other hardware resources, resulting in increased costs

Images