Unlock instant, AI-driven research and patent intelligence for your innovation.

Webpage data crawling method and device, computer equipment and storage medium

A web page data and data technology, applied in the field of data analysis, can solve the problem of low operation efficiency and achieve the effect of improving operation efficiency

Pending Publication Date: 2020-07-28
ONE CONNECT SMART TECH CO LTD SHENZHEN
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] It can be seen that the operating efficiency of the existing crawler program is not high when performing crawler tasks

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Webpage data crawling method and device, computer equipment and storage medium
  • Webpage data crawling method and device, computer equipment and storage medium
  • Webpage data crawling method and device, computer equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0048] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

[0049] A web page data crawling method provided by this application can be applied to such as figure 1 shown in the application environment. The crawling server 110 first executes the pre-delivered data crawling task; the data crawling task is used to instruct the crawling server 110 to call the corresponding target crawler program to perform a data crawling operation on the webpage to be crawled; when the crawling server 110 detects that the When crawling a webpage requires verification, trigger the webpage server to be crawled to send a verification code to the user ter...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a webpage data crawling method and device, computer equipment and a storage medium. The method comprises the following steps: executing a pre-issued data crawling task, wherein the data crawling task is used for indicating the crawling server to call a corresponding target crawler program to perform data crawling operation on a to-be-crawled webpage; when it is detected that the to-be-crawled webpage needs to be verified, triggering a webpage server of the to-be-crawled webpage to send a verification code to a user terminal, thereby enabling the user terminal to receive the verification code and upload the verification code to a background server for storage: determining a target verification code in a plurality of verification codes stored in the background serveraccording to a terminal identifier of the user terminal; and sending the target verification code to the webpage server for verification. The method relates to the field of general web crawlers, andby adopting the scheme, the running efficiency of an existing crawler program during crawler task execution can be improved.

Description

technical field [0001] The present application relates to the technical field of data analysis, and in particular to a web page data crawling method, device, computer equipment and storage medium. Background technique [0002] With the development of the Internet, we often need to verify the mobile phone verification code OTP (One TimePassword) when we log in to websites and apps, so that we can confirm personal identity when logging in to websites and apps, and reduce network hazards for user registration, login, and online payment. Protect users' personal information. [0003] When a developer or tester uses a crawler program for automatic operations such as data crawling, simulated login, and simulated application of the target website, the target website often requires filling in and verifying the SMS verification code in order to determine the real identity of the user. This also makes the developer or tester need to check the SMS verification code for the target websi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/951H04L29/06
CPCG06F16/951H04L63/145
Inventor 许剑勇
Owner ONE CONNECT SMART TECH CO LTD SHENZHEN