Unlock instant, AI-driven research and patent intelligence for your innovation.

Method and system for monitoring data crawling process

A data and data collection technology, applied in the monitoring field of the data crawling process, can solve the problems of low reliability of data crawling and failure to crawl data, etc., and achieve the goal of improving reliability, comprehensiveness and accuracy Effect

Pending Publication Date: 2021-03-09
CHINA INFORMATION TECH SECURITY EVALUATION CENT
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] However, the inventor found that when the crawler program is running normally, there will be a problem that the data cannot be crawled. Therefore, the above-mentioned detection method has limitations, resulting in low reliability of data crawling

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for monitoring data crawling process
  • Method and system for monitoring data crawling process
  • Method and system for monitoring data crawling process

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0047] In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0048] Such as figure 1 As shown, it is a flow chart of Embodiment 1 of a monitoring method for a data crawling process provided by the present application, and the method may include the following steps:

[0049] Step S11 , detecting ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a method and system for monitoring a data crawling process. The method is characterized in that the reason why a data crawling function is abnormal is determined by detecting whether the data crawling function is abnormal or not and detecting the running state of the crawler program, the accuracy of determining the reason why the data crawling function is abnormal can be improved, and the data crawling can be intervened according to the reason why the data crawling function is abnormal, and the reliability of the data crawling is improved. Moreover, when the running state of a crawler program is a normal state, a detection result is obtained by sending a detection message to a crawled object and detecting the network state of the crawled object, the website architecture adjustment state or the anti-crawling strategy starting state, and the reason why the data crawling function is abnormal is determined according to the detection result, so that the reason why thedata crawling function is abnormal is determined from the perspective except the crawler program, the comprehensiveness of the reason why the data crawling function is abnormal is improved, and the reliability of data crawling is further improved.

Description

technical field [0001] The present application relates to the technical field of data collection, and in particular to a monitoring method and system for a data crawling process. Background technique [0002] A crawler program is a program that automatically grabs information on the World Wide Web according to certain rules. At present, it is generally judged whether the data crawling is normal by detecting whether the crawler program is running normally, that is, if the crawler program is running normally, it is considered that the data crawling is normal. [0003] However, the inventors found that when the crawler program is running normally, there will be a problem that data cannot be crawled. Therefore, the above detection method has limitations, resulting in low reliability of data crawling. Contents of the invention [0004] In order to solve the above technical problems, the embodiment of the present application provides a monitoring method and system for the data ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/951G06F11/07G06F11/30
CPCG06F16/951G06F11/079G06F11/0766G06F11/302
Inventor 杨一未方硕易锦赵嘉伟
Owner CHINA INFORMATION TECH SECURITY EVALUATION CENT