Data collection method based on intelligent grasping system for science and technology service information

Abstract

The invention relates to a data collection method based on an intelligent grasping system for science and technology service information. The method comprises the following steps: 1, data grasping: in terms of configuration of a crawler, a user releases a grasping task through a configuration module and a start module of a client and sets a webpage for grasping and corresponding rules; 2, loading of timing grasping tasks: the task released by the user is loaded dynamically to a timing grasping task list; 3, page downloading; 4, page parsing: pages in a queue are parsed; 5, adding to URL to be grasped; 6, data processing and storage: page data is subjected to parsing extraction processing, and extracted two-dimensional structured data is stored. The method can meet the requirement of the crawler for generality and the requirement of the science and technology service system for grasping, and convenient extension and plug-in type development are realized; parser rule configuration, width and depth of the pages subjected to grasping, grasping threads, database configuration or index configuration is added to specific service logic, and information can be grasped and collected intelligently.

Description

technical field [0001] The invention relates to an intelligent capture method, in particular to a data collection method for an intelligent capture system based on scientific and technological service information. Background technique [0002] Crawler, also known as spider, is not the name of an insect, but a computer program that people use to continuously extract links to web pages through customized entry URLs on the Internet, and then crawl and extract deeper unknown links based on these links. Going forward, we describe such program crawling behavior as a crawler-like action, which is called a crawler. A crawler is a program that automatically obtains web content and is an important part of a search engine. [0003] The crawling system using artificially generated crawling wrapper technology has accurate scientific and technological information extraction results, but it is necessary to generate, update and maintain crawl wrappers for thousands of websites on the Intern...

AI Technical Summary

Problems solved by technology

[0003] The crawling system using artificially generated crawling wrapper technology has accurate scientific and technological information extraction results, but it is necessary to generate, update and maintain crawl wrappers for thousands of websites on the Internet. Ordinary vertical crawlers cannot do this well. Can rely on substantial human involvement

[0004] Safe and efficient real-time crawling technology; when high real-time crawling is required, it is necessary to frequently initiate link and download requests to the crawling website server, which will put a lot of pressure on the server of the other party, which in turn will cause the other party to use Banning policies such as denying access to ensure the normal operation of the server will lead to crawling failures; at the same time, high real-time crawling requirements consume a lot of network, server and other hardware resources, resulting in increased costs

[0005] With the continuous popularization of AJAX technology and the emergence of Single-page application frameworks such as AngularJS, more and more pages are now rendered by js; for crawlers, this kind of page is quite annoying: only extracting HTML content, It is often not possible to obtain valid information

Images