Method and device for acquiring web crawler request

A web crawler and acquisition method technology, applied in the field of web crawler request acquisition, can solve problems such as low web crawling efficiency, and achieve the effects of improving user experience and efficiency

Active Publication Date: 2017-11-21
BEIJING GRIDSUM TECH CO LTD
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The main purpose of the present invention is to provide a method and device for obtaining a web crawler request, to solve the problem of low web crawling efficiency caused by manual intervention in the verification code login process in the prior art to a certain extent

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for acquiring web crawler request
  • Method and device for acquiring web crawler request
  • Method and device for acquiring web crawler request

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029]It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and examples.

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0031] It should be noted that the terms "firs...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an acquisition method of a web crawler request and an acquisition device thereof. The acquisition method comprises the steps: acquiring verification code images of to-be-crawled resource information; performing division and noise reduction to the verification code images to obtain multiple first images; performing binarization to all first images to obtain the first matrixes of all first images; reading the second matrixes corresponding to the first matrixes from a database; acquiring characters indicated by the second matrixes to obtain verification code information; generating the web crawler request by the verification code information and the user information in advance acquired, wherein the web crawler request is used to acquire the resource information. Thus, through the method and the device, the low web crawling efficiency caused by manual intervention is partly solved in the existing technology, and the web crawling efficiency effect is improved.

Description

technical field [0001] The present invention relates to the field of the Internet, in particular to a method and device for obtaining web crawler requests. Background technique [0002] With the rapid development of the Internet, network resources are becoming more and more abundant, and human search for this information is far from meeting the needs. How to effectively extract and use this information has become a huge challenge. In order to solve the effective and intelligent extraction of information, web crawlers came into being. Before obtaining a specific resource, the web crawler first constructs the corresponding request header information and cookie settings to send a request to a specific site in the form of Post or Get, and then obtains the response resource information. However, the resource information of some sites on the Internet can only be obtained after logging in, and some sites even require the input of verification code information when logging in for t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
CPCG06F16/951
Inventor 李庆
Owner BEIJING GRIDSUM TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap